Using LLM Models for Text Classification: A Guide to Supervised Learning

#learnai #oxlo #ai

We are building a support ticket classifier that routes incoming messages to the right team using an LLM as a zero-shot classifier. I will walk you through labeling a small evaluation set, running classification through Oxlo.ai, and measuring accuracy against ground truth. This approach skips weeks of traditional model training and deploys in minutes.

What you'll need

Python 3.10 or newer
The OpenAI SDK: pip install openai
An Oxlo.ai API key from https://portal.oxlo.ai

Step 1: Configure the Oxlo.ai client

Instantiate the client pointing at Oxlo.ai. I use llama-3.3-70b as the general-purpose workhorse, but you can swap in qwen-3-32b for multilingual tickets or deepseek-v3.2 for coding-heavy queues.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "user", "content": "Confirm connectivity."},
    ],
)
print(response.choices[0].message.content)

Step 2: Define the system prompt and schema

The system prompt locks the model into a JSON-only response and defines the allowed categories. Keeping the schema rigid makes parsing trivial and accuracy reproducible.

SYSTEM_PROMPT = """You are a support ticket classifier.
Analyze the user message and return a JSON object with exactly two keys:
  category: one of Billing, Technical, Account, Feature Request
  confidence: a float between 0.0 and 1.0

Return only valid JSON. Do not wrap it in markdown."""

Step 3: Build the classify function with JSON mode

Oxlo.ai supports JSON mode, so we can force valid output structure instead of parsing free text. Because Oxlo.ai charges per request rather than per token, attaching long conversation history or large prompts does not inflate the cost of each classification call.

import json

def classify_ticket(text: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": text},
        ],
        response_format={"type": "json_object"},
    )
    raw = response.choices[0].message.content
    return json.loads(raw)

Step 4: Prepare a labeled evaluation set

Supervised learning needs ground truth. I created five examples that cover each category we care about. In production you would expand this to a few hundred examples, but this is enough to verify the pipeline.

EVAL_SET = [
    {
        "text": "I was charged twice for my subscription this month.",
        "true_label": "Billing",
    },
    {
        "text": "The API returns a 500 error when I POST to /v1/users.",
        "true_label": "Technical",
    },
    {
        "text": "I forgot my password and cannot reset it.",
        "true_label": "Account",
    },
    {
        "text": "Can you add dark mode to the dashboard?",
        "true_label": "Feature Request",
    },
    {
        "text": "My invoice shows the wrong tax amount.",
        "true_label": "Billing",
    },
]

Step 5: Run batch classification and compute accuracy

This loop calls Oxlo.ai for each ticket, compares the predicted category to the true label, and prints a summary. The platform has no cold starts on popular models, so the batch runs without warmup delays.

correct = 0
results = []

for item in EVAL_SET:
    prediction = classify_ticket(item["text"])
    predicted_label = prediction.get("category")
    is_match = predicted_label == item["true_label"]
    correct += int(is_match)
    results.append({
        "text_preview": item["text"][:40],
        "true": item["true_label"],
        "predicted": predicted_label,
        "confidence": prediction.get("confidence"),
        "match": is_match,
    })

for r in results:
    print(
        f"{r['text_preview']:<40} | True: {r['true']:<15} "
        f"| Pred: {r['predicted']:<15} | Conf: {r['confidence']}"
    )

print(f"\nAccuracy: {correct}/{len(EVAL_SET)} ({correct / len(EVAL_SET):.0%})")

Run it

Save the full script as ticket_classifier.py, export your key, and run it.

export OXLO_API_KEY="sk-..."
python ticket_classifier.py

Example output:

I was charged twice for my subscription th | True: Billing         | Pred: Billing         | Conf: 0.95
The API returns a 500 error when I POST  | True: Technical       | Pred: Technical       | Conf: 0.98
I forgot my password and cannot reset it | True: Account         | Pred: Account         | Conf: 0.97
Can you add dark mode to the dashboard?  | True: Feature Request | Pred: Feature Request | Conf: 0.94
My invoice shows the wrong tax amount.   | True: Billing         | Pred: Billing         | Conf: 0.96

Accuracy: 5/5 (100%)

Wrap-up

From here, add a confidence threshold so low-confidence predictions trigger human review, or expand the evaluation set to measure per-class F1 scores. If your tickets contain mixed languages, swap the model to qwen-3-32b on Oxlo.ai without changing any other code. For reasoning-heavy edge cases, try kimi-k2.6 or deepseek-r1-671b to see if chain-of-thought improves accuracy on ambiguous messages.

DEV Community