DEV Community

shashank ms
shashank ms

Posted on

Using LLM for Intent Recognition: A Comprehensive Guide

#learnai #oxlo #ai

We are going to build a production intent recognition pipeline that classifies incoming customer support messages into actionable categories. This helps support teams automatically route tickets to the right queue instead of relying on brittle keyword matching that breaks on ambiguous phrasing. I use Oxlo.ai for this because its request-based pricing means long ticket descriptions cost the same flat rate as short ones, which matters when user messages vary wildly in length.

What you'll need

Step 1: Set up the Oxlo.ai client and define the intent schema

Before writing any classification logic, I lock down the taxonomy. Every label must map to a concrete business action so the model has clear decision boundaries.

from openai import OpenAI
import json

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"  # grab this from https://portal.oxlo.ai
)

INTENTS = {
    "billing": "Questions about invoices, payments, refunds, or subscription changes.",
    "technical": "Bug reports, integration errors, API issues, or feature malfunctions.",
    "sales": "Pricing inquiries, demo requests, contract negotiations, or upgrade questions.",
    "general": "Any other request that does not fit the above categories."
}

Step 2: Write the system prompt that forces structured output

The system prompt is the entire product spec for the classifier. I force JSON mode through prompt engineering so I can parse the result without guessing.

SYSTEM_PROMPT = """You are an intent classification engine. Analyze the user's message and classify it into exactly one of the following intents: billing, technical, sales, general.

Definitions:
- billing: Questions about invoices, payments, refunds, or subscription changes.
- technical: Bug reports, integration errors, API issues, or feature malfunctions.
- sales: Pricing inquiries, demo requests, contract negotiations, or upgrade questions.
- general: Any other request that does not fit the above categories.

Respond ONLY with a JSON object in this exact format:
{
  "intent": "<one of the four labels>",
  "confidence": <integer between 1 and 10>,
  "reasoning": "<one sentence explaining why>"
}

Do not include markdown code fences or any text outside the JSON object."""

Step 3: Create the classify function with parsing and validation

I call Llama 3.3 70B on Oxlo.ai with a low temperature to keep the output deterministic. Then I strip accidental markdown fences and validate the intent against my schema.

def classify_intent(user_message: str, model: str = "llama-3.3-70b"):
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
        max_tokens=256,
    )
    
    raw = response.choices[0].message.content.strip()
    
    # Some models may wrap JSON in markdown, so strip fences if present
    if raw.startswith("

```"):
        raw = raw.split("\n", 1)[1].rsplit("```

", 1)[0].strip()
    
    result = json.loads(raw)
    
    if result["intent"] not in INTENTS:
        raise ValueError(f"Invalid intent returned: {result['intent']}")
    
    return result

Step 4: Build the router with confidence gating

Not every message deserves automatic routing. I add a confidence threshold so low confidence predictions fall back to a human queue rather than misfiring.

def route_ticket(user_message: str, min_confidence: int = 7):
    try:
        result = classify_intent(user_message)
    except Exception as e:
        return {
            "intent": "general",
            "confidence": 0,
            "reasoning": f"Parser/error fallback: {e}"
        }
    
    if result["confidence"] < min_confidence:
        result["intent"] = "general"
        result["reasoning"] += " (Confidence too low, routed to general.)"
    
    return result

Step 5: Process a batch of real messages

Here is how the pipeline looks in practice. I feed it a mix of billing, technical, sales, and vague praise to show how the boundaries hold up.

test_messages = [
    "My invoice last month was double charged and I need a refund immediately.",
    "Does the enterprise plan include SSO and can we schedule a demo?",
    "The webhook endpoint returns a 500 ever since your last update.",
    "Hey, just wanted to say your team rocks.",
]

for msg in test_messages:
    out = route_ticket(msg)
    print(f"Message: {msg[:50]}...")
    print(f"  -> intent: {out['intent']}, confidence: {out['confidence']}")
    print(f"  -> reasoning: {out['reasoning']}\n")

Run it

Running the script above against Oxlo.ai produces output similar to this:

Message: My invoice last month was double charged and I need...
  -> intent: billing, confidence: 9
  -> reasoning: The user explicitly mentions an invoice, double charge, and refund.

Message: Does the enterprise plan include SSO and can we schedule...
  -> intent: sales, confidence: 8
  -> reasoning: The user asks about plan features and requests a demo.

Message: The webhook endpoint returns a 500 ever since your last...
  -> intent: technical, confidence: 9
  -> reasoning: The user describes an API error and integration failure.

Message: Hey, just wanted to say your team rocks....
  -> intent: general, confidence: 6
  -> reasoning: Positive feedback with no actionable request. (Confidence too low, routed to general.)

Wrap-up and next steps

That is the core of a working intent router. To take it live, wire the route_ticket function into your helpdesk API or a Slack bot so tickets move automatically. If you need better accuracy on edge cases, add three to five few-shot examples to the system prompt using messages from your own logs. Oxlo.ai handles the long context without inflating cost, so expanding the prompt is cheap.

Top comments (0)