Using LLM for Text Classification with Transformers

#learnai #oxlo #ai

Introduction

We are building a support ticket classifier that routes incoming messages to the correct department using an LLM. This helps teams automate triage without maintaining a separate NLP pipeline or fine-tuning a model. Because Oxlo.ai charges a flat rate per request, the cost stays the same even when tickets include long logs or conversation threads, and you can see current plans at https://oxlo.ai/pricing.

What you'll need

Python 3.10 or newer installed locally.
An Oxlo.ai API key from https://portal.oxlo.ai.
The OpenAI SDK installed with pip install openai.

Step 1: Configure the Oxlo.ai client

Point the OpenAI SDK at Oxlo.ai and verify the connection with a lightweight smoke test. I always do this first so I know the key and endpoint are correct before I add any logic.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

response = client.chat.completions.create(
    model="deepseek-v3.2",
    messages=[{"role": "user", "content": "Say OK"}],
    max_tokens=10
)

print(response.choices[0].message.content)

Step 2: Define the classification schema

Our classifier will assign exactly one of four labels. Keeping the categories in a constant makes the prompt and any downstream routing logic easier to maintain.

CATEGORIES = ["Billing", "Technical", "Account", "Spam"]

print(f"Active categories: {CATEGORIES}")

Step 3: Write the system prompt

The system prompt is the only place where we teach the model the task. We force JSON output and ask for a confidence score so we can later filter low-certainty predictions.

SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the user's message and assign exactly one category from the following list:
Billing, Technical, Account, Spam.

Return only a JSON object with two keys:
- label: the assigned category
- confidence: an integer 0-100 representing certainty

Do not include markdown, explanations, or any text outside the JSON."""

Step 4: Build the classifier function

Now we wrap the API call in a small function. I set temperature to 0.1 to keep the model deterministic and use JSON mode so the output is parseable. I use llama-3.3-70b here because it follows structured instructions reliably, but you can swap in qwen-3-32b if you need multilingual tickets.

import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the user's message and assign exactly one category from the following list:
Billing, Technical, Account, Spam.

Return only a JSON object with two keys:
- label: the assigned category
- confidence: an integer 0-100 representing certainty

Do not include markdown, explanations, or any text outside the JSON."""

def classify_ticket(text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text}
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
        max_tokens=256
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

# Quick sanity check
print(classify_ticket("I was charged twice this month."))

Step 5: Run a batch over sample tickets

This is the complete script. We loop over a list of realistic tickets and print structured results. This same pattern works inside a worker, webhook handler, or CLI tool.

import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the user's message and assign exactly one category from the following list:
Billing, Technical, Account, Spam.

Return only a JSON object with two keys:
- label: the assigned category
- confidence: an integer 0-100 representing certainty

Do not include markdown, explanations, or any text outside the JSON."""

def classify_ticket(text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text}
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
        max_tokens=256
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

tickets = [
    "I was charged twice for my subscription this month. Please refund the extra payment.",
    "The API returns a 500 error every time I try to upload a batch larger than 100 items.",
    "Can you reset my password? I forgot it.",
    "Congratulations! You have won a free iPhone. Click here to claim."
]

for ticket in tickets:
    result = classify_ticket(ticket)
    print(f"{result['label']:12} | confidence {result['confidence']:3} | {ticket[:50]}...")

Run it

Save the script as ticket_classifier.py, replace YOUR_OXLO_API_KEY with your key, and execute it.

python ticket_classifier.py

Expected output:

Billing      | confidence  95 | I was charged twice for my subscription this month...
Technical    | confidence  92 | The API returns a 500 error every time I try to upl...
Account      | confidence  88 | Can you reset my password? I forgot it....
Spam         | confidence  98 | Congratulations! You have won a free iPhone. Click...

Next steps

Add a confidence threshold so any prediction below 80 is routed to a human reviewer instead of an automated queue. You can also swap the model to kimi-k2.6 if your tickets contain screenshots or mixed-language content.

DEV Community