Building Language Understanding Systems with LLM: A Step-by-Step Guide

We are building a support ticket triage agent that reads raw customer messages, classifies intent, extracts entities, and routes each ticket to the correct team queue. This kind of language understanding system replaces manual sorting and cuts first-response time. Because the system prompt includes full few-shot examples, Oxlo.ai's flat per-request pricing is a good fit here. Cost stays the same even when the prompt grows.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK installed with pip install openai

Step 1: Set up the client

I start a new file called triage.py and import the dependencies. Oxlo.ai is a drop-in replacement for the OpenAI client, so I only change the base_url.

from openai import OpenAI
import json
import re

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 2: Define the system prompt

This prompt is the contract. It lists the allowed intents, the entity types, and the exact JSON schema. I also embed two few-shot examples so the model stays consistent. With Oxlo.ai, adding these examples does not increase the per-request price.

SYSTEM_PROMPT = """You are a support ticket triage agent.
Read the user's message and return a single JSON object with these keys:
- intent: one of ["billing", "technical", "sales", "general"]
- urgency: one of ["low", "medium", "high", "critical"]
- entities: an object with any order_id, email, or product_name found in the text
- summary: one sentence describing the problem

Rules:
1. Output raw JSON only. No markdown code fences, no commentary.
2. An order_id starts with "ORD-" followed by digits.
3. If no entities are found, return an empty object.

Examples:
User: "I was charged twice for ORD-9981. Please refund me."
{"intent":"billing","urgency":"high","entities":{"order_id":"ORD-9981"},"summary":"Customer reports duplicate charge and requests refund."}

User: "How do I reset my password?"
{"intent":"technical","urgency":"medium","entities":{},"summary":"Customer needs password reset instructions."}
"""

Step 3: Build the understanding function

Now I wrap the API call. I use llama-3.3-70b because it handles structured instructions reliably. Oxlo.ai serves this model with no cold starts, so the first request after idle time is just as fast as any other.

def classify_ticket(user_message: str) -> dict:
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        max_tokens=512,
    )

    raw = response.choices[0].message.content.strip()

    # Defensive parse in case of leading whitespace or stray text
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        match = re.search(r'\{.*\}', raw, re.DOTALL)
        if match:
            return json.loads(match.group(0))
        raise ValueError(f"Unparseable output: {raw}")

Step 4: Add the routing layer

Classification is only useful if it drives an action. This function maps intent to an email queue and escalates anything marked critical.

def route_ticket(analysis: dict) -> str:
    intent = analysis.get("intent", "general")
    urgency = analysis.get("urgency", "low")
    summary = analysis.get("summary", "No summary.")

    queues = {
        "billing": "[e-mail]",
        "technical": "[e-mail]",
        "sales": "[e-mail]",
        "general": "[e-mail]",
    }

    destination = queues.get(intent, "[e-mail]")
    if urgency == "critical":
        destination = "[e-mail]"

    return json.dumps({
        "destination_queue": destination,
        "priority": urgency,
        "subject": f"[{intent.upper()}] {summary}",
        "extracted_entities": analysis.get("entities", {}),
    }, indent=2)

Step 5: Wire up the CLI

I add a small runner block to test three different messages end to end.

if __name__ == "__main__":
    samples = [
        "My ORD-5521 never arrived and I need it by tomorrow. This is critical.",
        "Do you offer discounts for nonprofit organizations?",
        "I keep getting a 500 error on the settings page.",
    ]

    for msg in samples:
        print("=" * 50)
        print("Input:", msg)
        analysis = classify_ticket(msg)
        print("Analysis:", json.dumps(analysis, indent=2))
        print("Routing:")
        print(route_ticket(analysis))
        print()

Run it

Save the file, swap in your real API key from https://portal.oxlo.ai, and run:

python triage.py

You should see JSON output similar to this:

==================================================
Input: My ORD-5521 never arrived and I need it by tomorrow. This is critical.
Analysis: {
  "intent": "billing",
  "urgency": "critical",
  "entities": {
    "order_id": "ORD-5521"
  },
  "summary": "Customer reports missing order ORD-5521 and needs it urgently."
}
Routing:
{
  "destination_queue": "[e-mail]",
  "priority": "critical",
  "subject": "[BILLING] Customer reports missing order ORD-5521 and needs it urgently.",
  "extracted_entities": {
    "order_id": "ORD-5521"
  }
}

==================================================
Input: Do you offer discounts for nonprofit organizations?
Analysis: {
  "intent": "sales",
  "urgency": "low",
  "entities": {},
  "summary": "Customer inquires about nonprofit discount availability."
}
Routing:
{
  "destination_queue": "[e-mail]",
  "priority": "low",
  "subject": "[SALES] Customer inquires about nonprofit discount availability.",
  "extracted_entities": {}
}

==================================================
Input: I keep getting a 500 error on the settings page.
Analysis: {
  "intent": "technical",
  "urgency": "high",
  "entities": {},
  "summary": "Customer experiences a 500 error on the settings page."
}
Routing:
{
  "destination_queue": "[e-mail]",
  "priority": "high",
  "subject": "[TECHNICAL] Customer experiences a 500 error on the settings page.",
  "extracted_entities": {}
}

Next steps

To productionize this, I would wrap the classifier in a FastAPI endpoint so other services can POST raw tickets and receive the routing JSON. If you start seeing ambiguous edge cases, swap the model string to kimi-k2.6 or deepseek-v3.2 for stronger reasoning without changing any other code. You can explore Oxlo.ai model options and flat request pricing at https://oxlo.ai/pricing.