Using LLM for Text Classification: Best Practices and Applications

#learnai #oxlo #ai

We are building a support ticket classifier that routes incoming messages to the correct team using an LLM. This is useful if you run a product with user feedback, support queues, or content moderation pipelines. I will walk through a small but production-ready Python service that classifies text with structured JSON output.

What you'll need

Python 3.10 or newer.
The OpenAI SDK: pip install openai.
An Oxlo.ai API key from https://portal.oxlo.ai. You can start on the free tier.

Step 1: Connect to Oxlo.ai and verify the endpoint

Before writing any logic, confirm that your API key and the Oxlo.ai endpoint are working. I use Llama 3.3 70B here because it follows instructions reliably for classification tasks, though you could also use Qwen 3 32B or DeepSeek V3.2.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Reply with exactly: Connection OK"}],
    max_tokens=10,
)
print(response.choices[0].message.content)

Step 2: Define the classification schema

Consistent output is critical for text classification. I keep the categories and JSON schema in constants so the model has no room for hallucinated labels.

import json

CATEGORIES = [
    "Billing",
    "Technical Bug",
    "Feature Request",
    "Account Access",
    "Spam",
]

SCHEMA = {
    "type": "object",
    "properties": {
        "category": {"type": "string", "enum": CATEGORIES},
        "confidence": {"type": "number", "minimum": 0, "maximum": 1},
        "reasoning": {"type": "string"},
    },
    "required": ["category", "confidence", "reasoning"],
}

Step 3: Write the system prompt

The system prompt acts as the classifier's rulebook. It lists the categories, gives short definitions, and mandates strict JSON output with no extra text.

SYSTEM_PROMPT = """You are a support ticket classifier.
Read the user's message and assign exactly one category from this list:
- Billing: questions about invoices, payments, refunds, or subscriptions.
- Technical Bug: unexpected errors, crashes, or broken functionality.
- Feature Request: ideas for new capabilities or improvements.
- Account Access: login issues, password resets, or 2FA problems.
- Spam: irrelevant or unsolicited content.

Respond ONLY with a JSON object matching the provided schema.
Do not wrap the JSON in markdown code fences.
Do not add explanation outside the JSON."""

Step 4: Build the classifier function

This function takes a raw ticket string, sends it to Oxlo.ai with the schema, and parses the result. I set temperature low to keep outputs deterministic, and I use JSON mode to guarantee valid JSON.

def classify_ticket(text: str) -> dict:
    user_content = (
        f"Classify the following support ticket.\n\n"
        f"Schema: {json.dumps(SCHEMA)}\n\n"
        f"Ticket:\n{text}"
    )

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_content},
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
        max_tokens=256,
    )

    raw = response.choices[0].message.content
    return json.loads(raw)

Step 5: Process a batch of tickets

Real pipelines handle more than one item. I loop over a list of sample tickets, classify each, and print a simple summary table.

tickets = [
    "I was charged twice for my Pro plan this month. Can I get a refund?",
    "The export to CSV button does nothing when I click it in Chrome.",
    "Please add dark mode to the dashboard. It hurts my eyes at night.",
    "I forgot my password and the reset email never arrives.",
    "Buy cheap watches now!!! Click here!!!",
]

for ticket in tickets:
    result = classify_ticket(ticket)
    cat = result["category"]
    conf = result["confidence"]
    preview = ticket[:50]
    print(f"{cat:15} | {conf:.2f} | {preview}...")

Run it

Save the script as ticket_classifier.py and run it. Your output should look similar to this.

$ python ticket_classifier.py
Billing         | 0.95 | I was charged twice for my Pro plan this month...
Technical Bug   | 0.92 | The export to CSV button does nothing when I cl...
Feature Request | 0.89 | Please add dark mode to the dashboard. It hurt...
Account Access  | 0.94 | I forgot my password and the reset email never...
Spam            | 0.99 | Buy cheap watches now!!! Click here!!!...

Wrap-up and next steps

You now have a working classifier that runs on Oxlo.ai's request-based pricing, which means long tickets do not inflate your cost the way token-based billing would. Two concrete next steps: wire the classifier into a webhook so incoming tickets route automatically, and add a confidence threshold that escalates low-confidence results to a human reviewer.

DEV Community