Optimizing LLMs for Text Classification: Best Practices and Use Cases

#engineering #oxlo #ai

We are going to build a production-ready support ticket classifier that reads incoming text and routes it to the correct department. I shipped a version of this last quarter to eliminate manual triage, and an LLM running on Oxlo.ai handles the heavy lifting without the cost surprises that come from token-based pricing on long tickets.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Wire up the Oxlo.ai client

I start every project by verifying the connection. This snippet initializes the OpenAI-compatible client against Oxlo.ai and makes a quick health check. I default to llama-3.3-70b for general classification, but you can swap in qwen-3-32b or deepseek-v3.2 on Oxlo.ai without changing any other code.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Say 'Connection to Oxlo.ai is working'"},
    ],
)
print(response.choices[0].message.content)

Step 2: Define the classification schema

Text classification only works if the model follows a strict output format. I use a system prompt that locks the response to JSON and defines the allowed categories. This keeps parsing logic simple and makes the output predictable.

SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the ticket and return a JSON object with exactly these keys:
- category: one of [Billing, Technical, Account_Management]
- reasoning: a short explanation of why you chose this category
Do not include markdown formatting or any text outside the JSON object."""

Step 3: Classify a single ticket with structured output

Now I wrap the call in a small function that sends the raw ticket body to the model and expects a JSON object back. I set response_format to json_object so the model knows not to add markdown or extra commentary.

import json

def classify_ticket(ticket_text: str):
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": ticket_text},
        ],
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

ticket = "I was charged twice for my subscription this month. Please refund the extra payment."
result = classify_ticket(ticket)
print(result)

Step 4: Calibrate with few-shot examples

Zero-shot classification is decent, but adding a few labeled examples in the conversation history improves accuracy on ambiguous phrasing. I prepend two examples before the real ticket so the model sees the exact pattern I want.

FEW_SHOTS = [
    {"role": "user", "content": "My server keeps returning 502 errors after the latest deploy."},
    {"role": "assistant", "content": '{"category": "Technical", "reasoning": "Infrastructure and error codes are technical issues."}'},
    {"role": "user", "content": "I need to add two more seats to our enterprise plan."},
    {"role": "assistant", "content": '{"category": "Account_Management", "reasoning": "Plan changes and seat counts are account management tasks."}'},
]

def classify_with_few_shot(ticket_text: str):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
    ] + FEW_SHOTS + [
        {"role": "user", "content": ticket_text},
    ]
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages,
        response_format={"type": "json_object"},
    )
    return json.loads(response.choices[0].message.content)

Step 5: Add confidence scoring and fallback handling

In production, I do not trust classifications below a confidence threshold. I update the schema to ask for a confidence score between 0.0 and 1.0, then wrap the classifier in logic that returns "needs_review" when the score is under 0.8. This prevents bad routing.

def classify_with_confidence(ticket_text: str):
    prompt = SYSTEM_PROMPT + "\nAlso include a confidence key with a float between 0.0 and 1.0."
    messages = [
        {"role": "system", "content": prompt},
    ] + FEW_SHOTS + [
        {"role": "user", "content": ticket_text},
    ]
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages,
        response_format={"type": "json_object"},
    )
    parsed = json.loads(response.choices[0].message.content)
    if parsed.get("confidence", 0) < 0.8:
        parsed["category"] = "needs_review"
    return parsed

Run it

Here is a small batch of real-looking tickets. The script loops through them, prints the classification, and flags anything uncertain.

tickets = [
    "I was charged twice for my subscription this month. Please refund the extra payment.",
    "How do I reset my password? I forgot it and the reset email is not arriving.",
    "We are evaluating your enterprise tier and need a SOC-2 report before procurement can sign off.",
    "URGENT: all our API calls are timing out after 30 seconds since 9 AM today.",
]

for t in tickets:
    out = classify_with_confidence(t)
    print(f"Ticket: {t[:50]}...")
    print(f"Result: {out}")
    print()

When I run this, the output looks like this:

Ticket: I was charged twice for my subscription this month...
Result: {'category': 'Billing', 'reasoning': 'Duplicate charges and refunds are billing issues.', 'confidence': 0.95}

Ticket: How do I reset my password? I forgot it and the re...
Result: {'category': 'Technical', 'reasoning': 'Password reset and email delivery are technical problems.', 'confidence': 0.91}

Ticket: We are evaluating your enterprise tier and need a ...
Result: {'category': 'needs_review', 'reasoning': 'Mixed enterprise sales and compliance request.', 'confidence': 0.72}

Ticket: URGENT: all our API calls are timing out after 30 ...
Result: {'category': 'Technical', 'reasoning': 'API timeouts indicate infrastructure issues.', 'confidence': 0.97}

Wrap-up

With the classifier working, the next logical step is to wire it into an async queue worker so tickets stream through Oxlo.ai in parallel. If your tickets often include long conversation threads, the flat per-request pricing at Oxlo.ai makes this pattern significantly cheaper than token-based providers because cost does not scale with input length. You can compare plans at https://oxlo.ai/pricing.

DEV Community