Using LLM for Text Classification: A Step-by-Step Guide

#learnai #oxlo #ai

Text classification is one of the fastest ways to put an LLM into production. In this guide, I will walk through building a lightweight support ticket classifier that labels incoming messages and returns structured JSON. It is aimed at teams that need to route, tag, or prioritize text without training and deploying a custom model.

What you'll need

Python 3.10 or newer
An Oxlo.ai API key from https://portal.oxlo.ai
The OpenAI SDK: pip install openai

Step 1: Configure the Oxlo.ai client

Oxlo.ai exposes a fully OpenAI-compatible endpoint, so the setup is a single client initialization. I will use the general-purpose Llama 3.3 70B model because it follows instructions reliably for structured outputs, and if your tickets are multilingual you can swap the model identifier to qwen-3-32b without changing any other code.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "user", "content": "Say 'Connection OK'"},
    ],
)
print(response.choices[0].message.content)

Step 2: Design the system prompt

The trick to consistent classification is giving the model a rigid system prompt with an explicit JSON schema. I treat the system prompt as config, not code, so it lives in its own block that you can edit without touching the logic.

SYSTEM_PROMPT = """You are a support ticket classifier. Given a user message, output a JSON object with exactly two keys:
- category: one of Billing, Technical, Account, Feature Request
- confidence: an integer from 1 to 10 representing your certainty

Rules:
1. Respond with only the JSON object, no markdown fences.
2. If the message is empty or unreadable, use category "Unknown" and confidence 1.
3. Base your decision only on the text provided."""

Step 3: Build the classifier function

Now I will wire the system prompt into a reusable function. It sends the ticket text to Oxlo.ai and parses the JSON response, with a fallback so a malformed payload does not crash the script.

import json

def classify_ticket(text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
        temperature=0.0,
    )
    raw = response.choices[0].message.content.strip()
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        return {"category": "Unknown", "confidence": 1, "raw": raw}

Step 4: Add batch processing and validation

Tickets usually arrive in groups, so I will add a small batch runner that classifies a list of strings and validates that every result contains the required keys.

from typing import List

def classify_batch(tickets: List[str]) -> List[dict]:
    results = []
    for ticket in tickets:
        result = classify_ticket(ticket)
        if "category" not in result or "confidence" not in result:
            result = {"category": "Unknown", "confidence": 1}
        results.append(result)
    return results

Run it

With the pieces in place, I will feed a few realistic support tickets through the pipeline and print the results.

if __name__ == "__main__":
    tickets = [
        "I was charged twice last month and need a refund.",
        "How do I reset my password? I cannot log in.",
        "Your API returns a 500 error every time I send a request with an empty body.",
        "Please add a dark mode option to the dashboard.",
        "asdfghjkl"
    ]

    classified = classify_batch(tickets)
    for text, res in zip(tickets, classified):
        print(f"Ticket: {text}")
        print(f"Result: {res}\n")

Example output:

Ticket: I was charged twice last month and need a refund.
Result: {'category': 'Billing', 'confidence': 9}

Ticket: How do I reset my password? I cannot log in.
Result: {'category': 'Account', 'confidence': 9}

Ticket: Your API returns a 500 error every time I send a request with an empty body.
Result: {'category': 'Technical', 'confidence': 8}

Ticket: Please add a dark mode option to the dashboard.
Result: {'category': 'Feature Request', 'confidence': 9}

Ticket: asdfghjkl
Result: {'category': 'Unknown', 'confidence': 1}

Wrap-up and next steps

Two concrete ways to push this forward. First, add a confidence threshold below which tickets are routed to a human reviewer, and test different cutoffs against a labeled holdout set to find the best threshold for your data. Second, replace the batch loop with an async queue worker so you can process tickets as they arrive in real time. Because Oxlo.ai uses flat per-request pricing, feeding the model a long conversation history for context does not inflate your bill the way token-based providers do. See https://oxlo.ai/pricing for details.

DEV Community