Using LLMs for Text Classification

#learnai #oxlo #ai

What we are building

We are building a zero-shot support ticket classifier that routes incoming text into predefined categories without any custom model training. This is ideal for teams that need to triage large volumes of user feedback, and because we will run it on Oxlo.ai, long tickets with stack traces or logs cost the same flat rate as short ones.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Set up the client and test a single classification

First, I will configure the OpenAI SDK to point at Oxlo.ai and send one ticket through Llama 3.3 70B to verify the setup and see the raw output.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

SYSTEM_PROMPT = """You are a support ticket classifier.
Classify the ticket into exactly one category:
- Billing
- Technical Issue
- Account Access
- Feature Request
- General Inquiry

Respond with only the category name, nothing else."""

ticket = "I was charged twice for my subscription this month. Please refund the extra payment."

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": ticket},
    ],
)

print(response.choices[0].message.content)

Step 2: Lock down categories in the system prompt

Hard-coding categories inside the prompt string gets messy fast. I will pull them into a Python list so I can reuse them in validation logic later, and keep the prompt itself clean.

CATEGORIES = [
    "Billing",
    "Technical Issue",
    "Account Access",
    "Feature Request",
    "General Inquiry",
]

SYSTEM_PROMPT = (
    "You are a support ticket classifier.\n"
    "Classify the ticket into exactly one category:\n"
    + "\n".join(f"- {c}" for c in CATEGORIES)
    + "\n\nRespond with only the category name, nothing else."
)

Step 3: Wrap the call in a classifier function

Now I will wrap the API call in a function that validates the model output against the allowed categories and falls back to "General Inquiry" if the model returns an unexpected string.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

CATEGORIES = [
    "Billing",
    "Technical Issue",
    "Account Access",
    "Feature Request",
    "General Inquiry",
]

SYSTEM_PROMPT = (
    "You are a support ticket classifier.\n"
    "Classify the ticket into exactly one category:\n"
    + "\n".join(f"- {c}" for c in CATEGORIES)
    + "\n\nRespond with only the category name, nothing else."
)

def classify_ticket(text: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
        temperature=0.0,
        max_tokens=20,
    )

    raw = response.choices[0].message.content.strip()

    for cat in CATEGORIES:
        if raw.lower() == cat.lower():
            return cat

    return "General Inquiry"

# Quick test
print(classify_ticket("How do I reset my password? I forgot it and the reset email is not arriving."))

Step 4: Classify a batch of tickets

In production you will handle many tickets at once. I will loop over a list, classify each one, and print a simple report. Because Oxlo.ai uses flat per-request pricing (https://oxlo.ai/pricing), a ticket with a long stack trace costs the same as a short question.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

CATEGORIES = [
    "Billing",
    "Technical Issue",
    "Account Access",
    "Feature Request",
    "General Inquiry",
]

SYSTEM_PROMPT = (
    "You are a support ticket classifier.\n"
    "Classify the ticket into exactly one category:\n"
    + "\n".join(f"- {c}" for c in CATEGORIES)
    + "\n\nRespond with only the category name, nothing else."
)

def classify_ticket(text: str) -> str:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
        temperature=0.0,
        max_tokens=20,
    )

    raw = response.choices[0].message.content.strip()

    for cat in CATEGORIES:
        if raw.lower() == cat.lower():
            return cat

    return "General Inquiry"

tickets = [
    "I was charged twice for my subscription this month. Please refund the extra payment.",
    "How do I reset my password? I forgot it and the reset email is not arriving.",
    "The API returns a 500 error when I send a request with an empty payload.",
    "It would be great if you added dark mode to the dashboard.",
    "Hi, I am wondering what your enterprise pricing looks like for a team of 50?",
]

for ticket in tickets:
    label = classify_ticket(ticket)
    print(f"{label:<20} | {ticket[:50]}...")

Run it

Save the full script from Step 4 as classify.py, replace YOUR_OXLO_API_KEY, and run it.

python classify.py

Example output:

Billing              | I was charged twice for my subscription this ...
Account Access       | How do I reset my password? I forgot it and th...
Technical Issue      | The API returns a 500 error when I send a requ...
Feature Request      | It would be great if you added dark mode to the...
General Inquiry      | Hi, I am wondering what your enterprise pricing...

Next steps

To make this production-ready, add a confidence threshold by asking the model to return a score from 1 to 10 and route low-confidence tickets to a human reviewer. You could also swap in qwen-3-32b or kimi-k2.6 on Oxlo.ai to compare latency and accuracy for your specific dataset without changing any other code.