When your support queue grows past a few hundred tickets a day, manual triage breaks down. In this tutorial, I will walk you through a lightweight pipeline that classifies incoming support tickets by urgency and category, then clusters them by underlying topic so you can spot emergent bugs before they escalate. We will build the whole thing on Oxlo.ai, taking advantage of flat per-request pricing so costs stay predictable even when ticket threads get long. See https://oxlo.ai/pricing for details.
What you'll need
Python 3.10 or newer. Install the OpenAI SDK and the data libraries we will use:
pip install openai pandas scikit-learnYou will also need an Oxlo.ai API key. Create one at https://portal.oxlo.ai.
Step 1: Set up the client and data
I start by initializing the OpenAI-compatible client pointing at Oxlo.ai and creating a small synthetic dataset so the script is runnable without any external files.
from openai import OpenAI
import pandas as pd
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
tickets = [
{"id": 1, "body": "Login page throws a 500 error after password entry. Cannot access dashboard."},
{"id": 2, "body": "How do I update my billing address? I moved to a new apartment last month."},
{"id": 3, "body": "The CSV export button disappeared from the analytics dashboard. This is blocking our monthly report."},
{"id": 4, "body": "Login fails on Safari after the latest update. Chrome works fine but Safari is broken."},
{"id": 5, "body": "Do you offer nonprofit discounts on annual plans?"},
{"id": 6, "body": "API calls time out with a 504 when I request more than 10,000 rows in a single batch."},
{"id": 7, "body": "Two-factor authentication SMS codes are delayed by ten minutes. Users cannot log in."},
{"id": 8, "body": "Can I downgrade from Pro to Free without losing historical data?"},
]
df = pd.DataFrame(tickets)
print(df)
Step 2: Classify tickets with structured outputs
Next, I define a system prompt that forces JSON mode. This removes parsing ambiguity. Then I call Llama 3.3 70B through Oxlo.ai for every ticket.
CLASSIFY_PROMPT = """You are a support triage assistant. Classify the ticket and respond ONLY with valid JSON.
Allowed categories: billing, bug, account, feature_request.
Allowed urgency levels: low, medium, high, critical.
JSON schema:
{"category": "...", "urgency": "...", "reason": "..."}
"""
import json
def classify_ticket(body: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": CLASSIFY_PROMPT},
{"role": "user", "content": body},
],
response_format={"type": "json_object"},
)
return json.loads(response.choices[0].message.content)
# Run classification
classifications = pd.json_normalize(df["body"].apply(classify_ticket))
df = pd.concat([df.reset_index(drop=True), classifications], axis=1)
print(df[["id", "category", "urgency", "reason"]])
Step 3: Embed and cluster
Classification gives us labels, but it does not surface hidden themes. I generate embeddings with BGE-Large and run KMeans to group similar tickets together.
import numpy as np
from sklearn.cluster import KMeans
def get_embedding(text: str) -> list[float]:
resp = client.embeddings.create(
model="bge-large",
input=text,
)
return resp.data[0].embedding
# Build the embedding matrix
df["embedding"] = df["body"].apply(get_embedding)
X = np.vstack(df["embedding"].to_numpy())
# Fit 3 clusters. Adjust n_clusters to your volume.
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
df["cluster_id"] = kmeans.fit_predict(X)
print(df[["id", "cluster_id"]])
Step 4: Label clusters
Cluster IDs are meaningless on their own. I sample a few tickets from each group and ask Qwen 3 32B to write a human-readable label and description.
CLUSTER_PROMPT = """You are a support analyst naming ticket themes. You will receive a few ticket excerpts from one cluster.
Respond ONLY with valid JSON in this exact shape:
{"label": "3-word label", "description": "one sentence describing the common issue"}
"""
def label_cluster(cluster_id: int) -> dict:
samples = df[df["cluster_id"] == cluster_id]["body"].head(5).tolist()
user_text = "\n".join(f"- {s}" for s in samples)
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": CLUSTER_PROMPT},
{"role": "user", "content": user_text},
],
response_format={"type": "json_object"},
)
return json.loads(response.choices[0].message.content)
# Map labels back
labels = {cid: label_cluster(cid) for cid in df["cluster_id"].unique()}
df["cluster_label"] = df["cluster_id"].map(lambda c: labels[c]["label"])
df["cluster_desc"] = df["cluster_id"].map(lambda c: labels[c]["description"])
print(df[["id", "cluster_label", "cluster_desc", "category", "urgency"]])
Run it
With the pipeline assembled, a single pass over the dataframe gives us classified, clustered, and labeled tickets. Here is the driver script and the output I get on the synthetic set.
if __name__ == "__main__":
for _, row in df.iterrows():
print(f"Ticket {row['id']} | {row['urgency']} | {row['category']}")
print(f" Cluster: {row['cluster_label']} - {row['cluster_desc']}")
print(f" Body: {row['body'][:80]}...")
print()
Example output:
Ticket 1 | critical | bug
Cluster: Authentication failures - Users experiencing login errors and 500 responses.
Body: Login page throws a 500 error after password entry. Cannot access dash...
Ticket 2 | low | billing
Cluster: Billing questions - Customers asking about address updates and pricing.
Body: How do I update my billing address? I moved to a new apartment last mo...
Ticket 3 | high | bug
Cluster: Data export issues - Reports of missing export buttons and API timeouts.
Body: The CSV export button disappeared from the analytics dashboard. This i...
Ticket 4 | high | bug
Cluster: Authentication failures - Users experiencing login errors and 500 responses.
Body: Login fails on Safari after the latest update. Chrome works fine but ...
Ticket 5 | low | billing
Cluster: Billing questions - Customers asking about address updates and pricing.
Body: Do you offer nonprofit discounts on annual plans?...
Ticket 6 | high | bug
Cluster: Data export issues - Reports of missing export buttons and API timeouts.
Body: API calls time out with a 504 when I request more than 10,000 rows i...
Ticket 7 | critical | bug
Cluster: Authentication failures - Users experiencing login errors and 500 responses.
Body: Two-factor authentication SMS codes are delayed by ten minutes. Users...
Ticket 8 | low | account
Cluster: Billing questions - Customers asking about address updates and pricing.
Body: Can I downgrade from Pro to Free without losing historical data?...
Next steps
Wire the classifier into your inbound email webhook so tickets are tagged on arrival. If you need to scale past a few thousand tickets, swap the synchronous loop for an async batch consumer, or add a vector cache so you do not re-embed duplicates.
Because Oxlo.ai charges per request rather than per token, adding the embedding call and the clustering label step does not explode costs when ticket threads get long. You can test the full pipeline against your own backlog without guessing the bill. See https://oxlo.ai/pricing for the latest plan details.
Top comments (0)