DEV Community

shashank ms
shashank ms

Posted on

Using LLMs for Text Classification Tasks: A Step-by-Step Guide

#learnai #oxlo #ai

This guide builds a working support ticket classifier that sorts inbound messages by category, priority, and sentiment using an LLM. It replaces the need to train a custom classification model, so you can deploy in minutes and change the schema later without gathering new training data.

What you'll need

Step 1: Connect to Oxlo.ai

Set up the client with your Oxlo.ai key and verify the connection with a quick ping. I use llama-3.3-70b here because it follows instructions reliably and starts instantly with no cold starts.

from openai import OpenAI
import os

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ.get("OXLO_API_KEY")
)

# Verify connectivity
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[{"role": "user", "content": "Reply with exactly: Connection OK"}],
    max_tokens=10
)
print(response.choices[0].message.content)

Step 2: Define the classification schema

The system prompt is the entire training step. It constrains the model to a JSON object with exactly the fields we need. Keep the category list short to reduce ambiguity.

SYSTEM_PROMPT = """You are a support ticket classifier. Read the ticket and return ONLY a JSON object with these exact keys:

- category: one of [Billing, Technical, Account, General]
- priority: one of [Low, Medium, High, Critical]
- sentiment: one of [Frustrated, Neutral, Satisfied]
- summary: a concise 10-word description of the issue

Rules:
- Use Critical only for outages, security incidents, or data loss.
- Output raw JSON. Do not wrap it in markdown code fences.
- If the ticket is empty or unreadable, set category to General and priority to Low."""

Step 3: Build the classifier function

This function sends the ticket to Oxlo.ai with JSON mode enabled so the model is constrained to valid JSON. Temperature is set low to keep outputs deterministic.

import json

def classify_ticket(ticket_text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Ticket:\n{ticket_text}"}
        ],
        response_format={"type": "json_object"},
        temperature=0.1,
        max_tokens=200
    )
    return json.loads(response.choices[0].message.content)

Step 4: Classify a batch of tickets

Now process several tickets in a loop. In production you might parallelize this, but a simple loop is enough to verify accuracy before you scale up.

tickets = [
    "I was charged twice this month and I need a refund immediately.",
    "How do I reset my password? I cannot log in anymore.",
    "Your API has been returning 500 errors since 9 AM.",
    "Thanks for the new dashboard widget, it saves me a lot of time."
]

results = []
for ticket in tickets:
    try:
        label = classify_ticket(ticket)
        label["ticket"] = ticket
        results.append(label)
        print(f"Done: {label['summary']}")
    except Exception as e:
        print(f"Failed on ticket: {e}")

# Inspect the first result
print(json.dumps(results[0], indent=2))

Run it

When you run the script, you should see output similar to this. The model consistently picks tight summaries and respects the category list.

Done: Duplicate charge refund request
Done: Password reset instructions needed
Done: API outage returning errors
Done: Positive feedback on dashboard widget

{
  "category": "Billing",
  "priority": "High",
  "sentiment": "Frustrated",
  "summary": "Duplicate charge refund request",
  "ticket": "I was charged twice this month and I need a refund immediately."
}

Wrap-up and next steps

You now have a classifier that you can adapt by editing the prompt, not by retraining a model. Two concrete next steps: wire this into your helpdesk webhook to label tickets on arrival, or add a confidence score by asking the model to include a confidence field and routing low-confidence items to a human reviewer.

Because Oxlo.ai uses flat per-request pricing, long conversation threads or lengthy attachments in tickets do not inflate your costs the way token-based pricing would. See https://oxlo.ai/pricing for details.

Top comments (0)