We are going to build a support ticket triage agent that uses in-context transfer learning to adapt a general-purpose LLM to a company-specific support taxonomy. By embedding labeled examples into the system prompt, we transfer domain knowledge without managing any training infrastructure. Because Oxlo.ai charges a flat rate per request, iterating on long prompts full of examples costs the same as a one-line query, which makes this approach cheap to test and deploy.
What you'll need
- Python 3.10 or higher
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK installed with
pip install openai
1. Configure the Oxlo.ai client
I always verify the endpoint before I add any logic. Create a client pointing to Oxlo.ai and run a quick health check with Llama 3.3 70B to confirm your key and the route are working.
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY" # from https://portal.oxlo.ai
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "Say OK"}],
max_tokens=10,
)
print(response.choices[0].message.content)2. Gather transfer examples
Transfer learning needs data. I define the target taxonomy and collect three representative examples that cover different labels and urgency levels. I store them as constants so I can inject them into the prompt cleanly.
CATEGORIES = [
"billing-refund",
"product-question",
"technical-bug",
"account-access",
"security-report",
]
FEW_SHOT_EXAMPLES = [
{
"ticket": "I was charged twice for my subscription this month.",
"output": '{"label": "billing-refund", "urgency": "high", "reply": "I have flagged the duplicate charge for review. Refunds typically process within 3 business days."}'
},
{
"ticket": "How do I export my data to CSV?",
"output": '{"label": "product-question", "urgency": "low", "reply": "You can export from Settings > Data > Export. CSV and JSON formats are supported."}'
},
{
"ticket": "The API returns a 500 error when I post to /v1/batch.",
"output": '{"label": "technical-bug", "urgency": "high", "reply": "Thanks for the report. I am escalating this to engineering and will update you within 30 minutes."}'
},
]3. Build the system prompt
This is the core transfer learning step. I embed the examples directly into the system prompt to adapt Llama 3.3 70B from general reasoning to our exact taxonomy and tone. The prompt is long, but on Oxlo.ai the cost is flat per request, so these extra tokens do not increase the price.
SYSTEM_PROMPT = """You are a support ticket triage agent for a B2B SaaS platform.
Analyze the incoming ticket, classify it, assess urgency, and draft a first response.
Rules:
- Output ONLY a JSON object with keys: label, urgency, reply.
- label must be one of: billing-refund, product-question, technical-bug, account-access, security-report.
- urgency must be one of: low, medium, high, critical.
- reply should be polite, concise, and accurate.
Here are examples of correct behavior:
Example 1:
Ticket: I was charged twice for my subscription this month.
Output: {"label": "billing-refund", "urgency": "high", "reply": "I have flagged the duplicate charge for review. Refunds typically process within 3 business days."}
Example 2:
Ticket: How do I export my data to CSV?
Output: {"label": "product-question", "urgency": "low", "reply": "You can export from Settings > Data > Export. CSV and JSON formats are supported."}
Example 3:
Ticket: The API returns a 500 error when I post to /v1/batch.
Output: {"label": "technical-bug", "urgency": "high", "reply": "Thanks for the report. I am escalating this to engineering and will update you within 30 minutes."}
Now process the user's ticket and return only the JSON object."""4. Wire the inference call
Now I connect the prompt to the Oxlo.ai chat completions endpoint. I keep the temperature low because classification tasks need deterministic outputs. I also strip any accidental markdown fences from the model response before parsing JSON.
import json
def triage_ticket(ticket_text: str) -> dict:
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": f"Ticket: {ticket_text}"},
],
temperature=0.1,
max_tokens=256,
)
raw = response.choices[0].message.content.strip()
# Some models wrap JSON in markdown fences; strip them if present.
if raw.startswith("
```"):
raw = raw.replace("```
json", "").replace("
```
", "").strip()
return json.loads(raw)5. Add guardrails
I add a small retry loop to handle malformed JSON or hallucinated labels. If the model still fails after two tries, I fall back to a safe default so the pipeline never crashes in production.
import time
def triage_with_retry(ticket_text: str, retries: int = 2) -> dict:
for attempt in range(retries + 1):
try:
result = triage_ticket(ticket_text)
assert result.get("label") in CATEGORIES
assert result.get("urgency") in ["low", "medium", "high", "critical"]
return result
except Exception:
if attempt == retries:
return {
"label": "unknown",
"urgency": "medium",
"reply": "A human agent will review this shortly."
}
time.sleep(1)
# Sanity check with an out-of-sample ticket.
print(triage_with_retry("I forgot my password and the reset email never arrives."))Run it
I run the agent over a small batch of unseen tickets. The transfer learned taxonomy holds, and the model generalizes to new phrasing while keeping the JSON schema intact.
if __name__ == "__main__":
tickets = [
"I forgot my password and the reset email never arrives.",
"There is a security vulnerability in your auth flow.",
"Do you offer annual pricing discounts?",
]
for t in tickets:
out = triage_with_retry(t)
print(f"Ticket: {t}")
print(f"Result: {json.dumps(out, indent=2)}")
print()Example output:
Ticket: I forgot my password and the reset email never arrives.
Result: {
"label": "account-access",
"urgency": "high",
"reply": "I can help with that. Please check your spam folder first, then let me know if you need a manual reset link."
}
Ticket: There is a security vulnerability in your auth flow.
Result: {
"label": "security-report",
"urgency": "critical",
"reply": "Thank you for the responsible disclosure. I am routing this to our security team immediately."
}
Ticket: Do you offer annual pricing discounts?
Result: {
"label": "product-question",
"urgency": "low",
"reply": "Yes, annual plans include a 15 percent discount. I can apply it to your account if you decide to switch."
}Wrap-up and next steps
Expand the few-shot set to ten or twenty examples and run a held-out evaluation to measure per-label accuracy. If you need stronger reasoning for ambiguous edge cases, swap the model string to qwen-3-32b or kimi-k2.6 on Oxlo.ai without touching any other client code, and the flat per-request pricing means the longer prompt still costs the same as before. You can review the latest plans at https://oxlo.ai/pricing.
Top comments (0)