Using LLM for Text Classification with Transformers: Best Practices

We are going to build a production-ready support ticket classifier that categorizes unstructured text into predefined labels using an LLM. I will use Oxlo.ai as the inference backend because its request-based pricing keeps costs flat even when tickets contain long conversation threads, and the OpenAI-compatible API means we can use JSON mode with zero client changes. This tutorial is for engineers who want to replace a traditional ML pipeline with a maintainable prompt-based system.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK installed with pip install openai

Step 1: Configure the Oxlo.ai client

I start by instantiating the OpenAI SDK against Oxlo.ai's endpoint. This is a literal drop-in replacement: only the base URL and API key change.

from openai import OpenAI
import json
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

# Verify connectivity
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Say OK"},
    ],
    max_tokens=5
)
print(response.choices[0].message.content)

Step 2: Define the classification schema and system prompt

The system prompt is the only "training" we need. It constrains the model to a strict JSON schema with allowed categories and a confidence score, so downstream code can parse the output without regex.

SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the user's message and return a JSON object with exactly these keys:
- category: one of ["Billing", "Technical", "Account", "General"]
- confidence: a float between 0.0 and 1.0
- reasoning: one sentence explaining why you chose this category

Rules:
- Return only the JSON object, with no markdown formatting.
- If the ticket mentions payment, invoices, or charges, use Billing.
- If it mentions bugs, errors, or integrations, use Technical.
- If it mentions login, password, or user settings, use Account.
- Use General only if none of the above apply."""

Step 3: Build the classifier function using JSON mode

Oxlo.ai supports the OpenAI SDK's JSON mode via response_format, which forces valid JSON output. I keep temperature low to reduce variance across repeated classifications.

def classify_ticket(text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Classify this support ticket:\n\n{text}"},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
        max_tokens=256
    )
    
    raw = response.choices[0].message.content
    return json.loads(raw)

# Test on a short example
sample = "I was charged twice for my Pro plan this month. Please refund the extra payment."
result = classify_ticket(sample)
print(json.dumps(result, indent=2))

Step 4: Evaluate against a small labeled dataset

Before deploying, I sanity-check the classifier on a handful of known examples. This catches prompt drift without requiring a heavy test harness.

test_set = [
    ("My credit card was declined but I still got charged.", "Billing"),
    ("The API returns a 500 error when I send large payloads.", "Technical"),
    ("I forgot my password and the reset email never arrives.", "Account"),
    ("What are your office hours?", "General"),
    ("Can I downgrade from Premium to Pro mid-cycle?", "Billing"),
]

correct = 0
for text, expected in test_set:
    prediction = classify_ticket(text)
    predicted_label = prediction.get("category", "Unknown")
    status = "PASS" if predicted_label == expected else "FAIL"
    if status == "PASS":
        correct += 1
    print(f"{status}: expected {expected}, got {predicted_label} (confidence: {prediction.get('confidence')})")

print(f"\nAccuracy: {correct}/{len(test_set)} ({correct/len(test_set):.0%})")

Step 5: Add confidence thresholds and batch processing

In production, low-confidence predictions should be flagged for human review. I also process tickets in a simple loop, which stays fast because Oxlo.ai serves popular models with no cold starts.

def classify_with_fallback(text: str, threshold: float = 0.85) -> dict:
    result = classify_ticket(text)
    confidence = float(result.get("confidence", 0.0))
    
    if confidence < threshold:
        result["category"] = "HumanReview"
        result["reasoning"] = f"Confidence {confidence:.2f} below threshold {threshold}"
    
    return result

def batch_classify(tickets: list[str]) -> list[dict]:
    results = []
    for ticket in tickets:
        try:
            results.append(classify_with_fallback(ticket))
        except Exception as e:
            results.append({"category": "Error", "error": str(e)})
    return results

long_tickets = [
    "I noticed something strange on my invoice. " * 20 + "Can you help?",
    "The webhook keeps timing out after 30 seconds. " * 20 + "Is there a fix?",
]

batch_results = batch_classify(long_tickets)
for r in batch_results:
    print(r["category"], "-", r.get("confidence") or r.get("error"))

Run it

Save the finished script as classifier.py, export your OXLO_API_KEY, and run it against a new ticket. Here is the end-to-end call:

import os
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the user's message and return a JSON object with exactly these keys:
- category: one of ["Billing", "Technical", "Account", "General"]
- confidence: a float between 0.0 and 1.0
- reasoning: one sentence explaining why you chose this category

Rules:
- Return only the JSON object, with no markdown formatting.
- If the ticket mentions payment, invoices, or charges, use Billing.
- If it mentions bugs, errors, or integrations, use Technical.
- If it mentions login, password, or user settings, use Account.
- Use General only if none of the above apply."""

def classify_ticket(text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Classify this support ticket:\n\n{text}"},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
        max_tokens=256
    )
    return json.loads(response.choices[0].message.content)

if __name__ == "__main__":
    ticket = "My SSO integration stopped working after the latest update. Users cannot log in through Okta."
    out = classify_ticket(ticket)
    print(json.dumps(out, indent=2))

Example output:

{
  "category": "Technical",
  "confidence": 0.94,
  "reasoning": "The ticket describes an SSO integration failure and login issue caused by a software update, which falls under technical problems."
}

Next steps

If you need hierarchical or multi-label classification, swap in Qwen 3 32B or DeepSeek V3.2 from Oxlo.ai's model catalog. The flat per-request pricing becomes a major advantage here, because long tickets or threaded conversations do not inflate your bill the way token-based providers do. See https://oxlo.ai/pricing for details.

A concrete next step is to pipe every low-confidence prediction into a review queue, then feed those human-verified labels back into your system prompt as few-shot examples. This closed loop will tighten accuracy without touching any model weights.