DEV Community

shashank ms
shashank ms

Posted on

Using LLM for Text Classification

#learnai #oxlo #ai

We are building a zero-shot text classifier that sorts customer support tickets into categories like Billing, Technical, Account, or General. This is useful when you need classification without maintaining a fine-tuned model or labeled training data. I use Oxlo.ai because its request-based pricing keeps the cost flat even when tickets are long, and the OpenAI-compatible SDK means no new client libraries to learn.

What you'll need

Step 1: Configure the Oxlo.ai client

I start by importing the SDK and pointing it at Oxlo.ai's endpoint. Because Oxlo.ai is fully OpenAI-compatible, this is a drop-in replacement.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY", "YOUR_OXLO_API_KEY")
)

Step 2: Write the classification system prompt

The prompt constrains the model to return only a JSON object with the predicted category and a short reason. This makes parsing trivial.

SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the user's message and classify it into exactly one of these categories:
- Billing
- Technical
- Account
- General

Respond with a JSON object containing two keys:
- "category": the chosen category
- "reason": one sentence explaining why

Do not include markdown formatting, only raw JSON."""

Step 3: Create the classifier function

This function sends the ticket text to Oxlo.ai and parses the JSON response. I use Llama 3.3 70B because it follows structured instructions reliably.

import json

def classify_ticket(text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
    )
    
    raw = response.choices[0].message.content.strip()
    return json.loads(raw)

# Quick sanity check
if __name__ == "__main__":
    test = "I was charged twice for my subscription last month."
    print(classify_ticket(test))

Step 4: Process a batch of tickets

In production you will classify more than one ticket. I iterate over a list, call the classifier for each, and collect results.

tickets = [
    "I was charged twice for my subscription last month.",
    "The API returns a 504 timeout every time I upload a file larger than 100 MB.",
    "I forgot my password and the reset email never arrives.",
    "Do you offer discounts for nonprofit organizations?"
]

results = []
for ticket in tickets:
    try:
        out = classify_ticket(ticket)
        results.append({
            "ticket": ticket,
            "category": out["category"],
            "reason": out["reason"]
        })
    except Exception as e:
        results.append({"ticket": ticket, "error": str(e)})

for r in results:
    print(r)

Step 5: Measure accuracy against a labeled test set

To verify the classifier works, I compare predictions against a small labeled dataset and compute accuracy.

labeled = [
    ("I was charged twice for my subscription last month.", "Billing"),
    ("The API returns a 504 timeout every time I upload a file larger than 100 MB.", "Technical"),
    ("I forgot my password and the reset email never arrives.", "Account"),
    ("Do you offer discounts for nonprofit organizations?", "General"),
]

correct = 0
for text, expected in labeled:
    try:
        pred = classify_ticket(text)["category"]
        if pred == expected:
            correct += 1
        print(f"Expected: {expected} | Predicted: {pred}")
    except Exception as e:
        print(f"Expected: {expected} | Error: {e}")

print(f"\nAccuracy: {correct}/{len(labeled)} ({correct / len(labeled):.0%})")

Run it

Save the script as classify.py, set your API key, and run it. Here is what the batch output looks like on my machine.

$ export OXLO_API_KEY=oxlo.ai-...
$ python classify.py

{'ticket': 'I was charged twice for my subscription last month.', 'category': 'Billing', 'reason': 'The user explicitly mentions being charged twice, which is a billing issue.'}
{'ticket': 'The API returns a 504 timeout every time I upload a file larger than 100 MB.', 'category': 'Technical', 'reason': 'The user describes an API timeout error during file uploads, indicating a technical problem.'}
{'ticket': 'I forgot my password and the reset email never arrives.', 'category': 'Account', 'reason': 'The user is experiencing issues with password reset and email delivery related to their account.'}
{'ticket': 'Do you offer discounts for nonprofit organizations?', 'category': 'General', 'reason': 'The user is asking about general pricing policies, not reporting a specific billing, technical, or account issue.'}

Accuracy: 4/4 (100%)

Next steps

You can extend this by adding confidence scores or mapping categories directly to team Slack channels. If you process high volumes of long tickets, Oxlo.ai's per-request pricing keeps costs flat regardless of input length. You can also swap in Qwen 3 32B or Kimi K2.6 from Oxlo.ai if you need stronger multilingual or reasoning performance.

Top comments (0)