I recently shipped an internal support triage bot to cut down on on-call noise. It classifies incoming tickets, drafts an internal note, and routes them to the right team. Along the way I hit the same traps that stall most LLM projects, so this tutorial walks through the working agent I ended up with, one pitfall at a time.
What you'll need
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK and a few helpers:
pip install openai pydantic tenacity
Step 1: Fix the system prompt first
My first draft used a vague system prompt and the model alternated between markdown, JSON, and plain sentences. The fix is a rigid prompt that defines the persona, output rules, and edge cases up front.
import os
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
SYSTEM_PROMPT = """You are a support ticket triage agent.
Your job is to read a customer ticket and produce a structured analysis.
Rules:
- Classify urgency as low, medium, or high.
- Assign to exactly one team: billing, technical, or account-management.
- Write a concise internal note summarizing the issue and suggested next step.
- Set category to spam if the message lacks a concrete support request.
- Respond with valid JSON containing these keys: urgency, team, category, internal_note, confidence.
- confidence must be a float between 0.0 and 1.0 representing your certainty."""
ticket = "I was charged twice for my subscription this month. Please fix this immediately."
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": ticket},
],
)
print(response.choices[0].message.content)
Step 2: Enforce structure with JSON mode
Parsing free text with regex is the second classic pitfall. I switched to JSON mode and added a Pydantic model so any shape mismatch fails fast before it reaches our routing logic.
import os
import json
from openai import OpenAI
from pydantic import BaseModel, Field
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
class TicketAnalysis(BaseModel):
urgency: str = Field(pattern="^(low|medium|high)$")
team: str = Field(pattern="^(billing|technical|account-management)$")
category: str = Field(pattern="^(spam|actionable)$")
internal_note: str
confidence: float = Field(ge=0.0, le=1.0)
SYSTEM_PROMPT = """You are a support ticket triage agent.
Your job is to read a customer ticket and produce a structured analysis.
Rules:
- Classify urgency as low, medium, or high.
- Assign to exactly one team: billing, technical, or account-management.
- Write a concise internal note summarizing the issue and suggested next step.
- Set category to spam if the message lacks a concrete support request.
- Respond with valid JSON containing these keys: urgency, team, category, internal_note, confidence.
- confidence must be a float between 0.0 and 1.0 representing your certainty."""
ticket = "I was charged twice for my subscription this month. Please fix this immediately."
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Analyze this ticket and return JSON:\n\n{ticket}"},
],
response_format={"type": "json_object"},
)
parsed = TicketAnalysis.model_validate_json(response.choices[0].message.content)
print(parsed.model_dump_json(indent=2))
Step 3: Plan for long context
The third pitfall is blind truncation. I used to slice logs to stay under a token budget. Because Oxlo.ai uses flat per-request pricing, cost does not scale with input length, so I stopped pre-truncating and started sending full threads. I still cap at a safe character limit to respect the model's context window, but cost is no longer the reason.
import os
import json
from openai import OpenAI
from pydantic import BaseModel, Field
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
class TicketAnalysis(BaseModel):
urgency: str = Field(pattern="^(low|medium|high)$")
team: str = Field(pattern="^(billing|technical|account-management)$")
category: str = Field(pattern="^(spam|actionable)$")
internal_note: str
confidence: float = Field(ge=0.0, le=1.0)
SYSTEM_PROMPT = """You are a support ticket triage agent.
Your job is to read a customer ticket and produce a structured analysis.
Rules:
- Classify urgency as low, medium, or high.
- Assign to exactly one team: billing, technical, or account-management.
- Write a concise internal note summarizing the issue and suggested next step.
- Set category to spam if the message lacks a concrete support request.
- Respond with valid JSON containing these keys: urgency, team, category, internal_note, confidence.
- confidence must be a float between 0.0 and 1.0 representing your certainty."""
def analyze_ticket(ticket_text: str) -> TicketAnalysis:
MAX_CHARS = 100000
if len(ticket_text) > MAX_CHARS:
ticket_text = ticket_text[:MAX_CHARS] + "\n[truncated]"
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Analyze this ticket and return JSON:\n\n{ticket_text}"},
],
response_format={"type": "json_object"},
)
return TicketAnalysis.model_validate_json(response.choices[0].message.content)
ticket = "I was charged twice for my subscription this month. Please fix this immediately."
print(analyze_ticket(ticket).model_dump_json(indent=2))
Step 4: Add retries and timeouts
A single timeout or rate-limit error should not drop a ticket. I wrap every call in tenacity so transient errors retry automatically.
import os
import json
from openai import OpenAI
from pydantic import BaseModel, Field
from tenacity import retry, stop_after_attempt, wait_exponential
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
class TicketAnalysis(BaseModel):
urgency: str = Field(pattern="^(low|medium|high)$")
team: str = Field(pattern="^(billing|technical|account-management)$")
category: str = Field(pattern="^(spam|actionable)$")
internal_note: str
confidence: float = Field(ge=0.0, le=1.0)
SYSTEM_PROMPT = """You are a support ticket triage agent.
Your job is to read a customer ticket and produce a structured analysis.
Rules:
- Classify urgency as low, medium, or high.
- Assign to exactly one team: billing, technical, or account-management.
- Write a concise internal note summarizing the issue and suggested next step.
- Set category to spam if the message lacks a concrete support request.
- Respond with valid JSON containing these keys: urgency, team, category, internal_note, confidence.
- confidence must be a float between 0.0 and 1.0 representing your certainty."""
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def call_oxlo(messages, response_format=None):
return client.chat.completions.create(
model="qwen-3-32b",
messages=messages,
response_format=response_format,
)
def analyze_ticket(ticket_text: str) -> TicketAnalysis:
MAX_CHARS = 100000
if len(ticket_text) > MAX_CHARS:
ticket_text = ticket_text[:MAX_CHARS] + "\n[truncated]"
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Analyze this ticket and return JSON:\n\n{ticket_text}"},
]
response = call_oxlo(messages, response_format={"type": "json_object"})
return TicketAnalysis.model_validate_json(response.choices[0].message.content)
ticket = "I was charged twice for my subscription this month. Please fix this immediately."
print(analyze_ticket(ticket).model_dump_json(indent=2))
Step 5: Validate outputs before you act
The last pitfall is treating the LLM like a deterministic API. Even with JSON mode, confidence scores can be low. I gate anything under 0.7 or any high-urgency ticket for human review.
import os
import json
from openai import OpenAI
from pydantic import BaseModel, Field
from tenacity import retry, stop_after_attempt, wait_exponential
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.getenv("OXLO_API_KEY")
)
class TicketAnalysis(BaseModel):
urgency: str = Field(pattern="^(low|medium|high)$")
team: str = Field(pattern="^(billing|technical|account-management)$")
category: str = Field(pattern="^(spam|actionable)$")
internal_note: str
confidence: float = Field(ge=0.0, le=1.0)
SYSTEM_PROMPT = """You are a support ticket triage agent.
Your job is to read a customer ticket and produce a structured analysis.
Rules:
- Classify urgency as low, medium, or high.
- Assign to exactly one team: billing, technical, or account-management.
- Write a concise internal note summarizing the issue and suggested next step.
- Set category to spam if the message lacks a concrete support request.
- Respond with valid JSON containing these keys: urgency, team, category, internal_note, confidence.
- confidence must be a float between 0.0 and 1.0 representing your certainty."""
@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
def call_oxlo(messages, response_format=None):
return client.chat.completions.create(
model="qwen-3-32b",
messages=messages,
response_format=response_format,
)
def analyze_ticket(ticket_text: str) -> TicketAnalysis:
MAX_CHARS = 100000
if len(ticket_text) > MAX_CHARS:
ticket_text = ticket_text[:MAX_CHARS] + "\n[truncated]"
messages = [
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Analyze this ticket and return JSON:\n\n{ticket_text}"},
]
response = call_oxlo(messages, response_format={"type": "json_object"})
return TicketAnalysis.model_validate_json(response.choices[0].message.content)
def triage_ticket(ticket_text: str) -> dict:
analysis = analyze_ticket(ticket_text)
result = {
"urgency": analysis.urgency,
"team": analysis.team,
"internal_note": analysis.internal_note,
"confidence": analysis.confidence,
"requires_human_review": False,
}
if analysis.confidence < 0.7 or analysis.urgency == "high":
result["requires_human_review"] = True
return result
ticket = "I was charged twice for my subscription this month. Please fix this immediately."
print(json.dumps(triage_ticket(ticket), indent=2))
The system prompt
After tuning against a hundred production tickets, this is the prompt I landed on. It is specific enough to remove ambiguity but short enough to minimize injection surface area.
SYSTEM_PROMPT = """You are a support ticket triage agent.
Your job is to read a customer ticket and produce a structured analysis.
Rules:
- Classify urgency as low, medium, or high.
- Assign to exactly one team: billing, technical, or account-management.
- Write a concise internal note summarizing the issue and suggested next step.
- Set category to spam if the message lacks a concrete support request.
- Respond with valid JSON containing these keys: urgency, team, category, internal_note, confidence.
- confidence must be a float between 0.0 and 1.0 representing your certainty."""
Run it
This script wires everything together. Export your Oxlo.ai key and run the full flow on a realistic ticket.
if __name__ == "__main__":
ticket = """Subject: Urgent - API returning 502s since 09:00 UTC
Our production webhook endpoint has been failing since this morning.
Every POST to /v1/events returns a 502 Bad Gateway.
This is blocking our checkout flow. We need an ETA on the fix immediately.
- Account: ACME Corp
- Region: us-east-1
- Error rate: 100% since 09:00 UTC
"""
decision = triage_ticket(ticket)
print(json.dumps(decision, indent=2))
Example output:
{
"urgency": "high",
"team": "technical",
"internal_note": "Customer reports total outage on POST /v1/events since 09:00 UTC. Route to infrastructure on-call immediately and provide customer ETA.",
"confidence": 0.96,
"requires_human_review": true
}
Next steps
Wire this function into your ticketing system's inbound webhook so it runs on every new message. If you want to experiment with different model behaviors, swap in Llama 3.3 70B for general-purpose accuracy or DeepSeek V3.2 for coding-heavy tickets. Both are available on Oxlo.ai with the same flat per-request pricing and no cold starts, so you can A/B without reworking your cost model.
Top comments (0)