Building a Customer Support Agent with LLM and NLU

#product #oxlo #ai

Customer support agents built on large language models need more than prompt engineering to survive production. They need a structured natural language understanding layer to classify intent, extract entities, and gatekeep sensitive actions before the LLM generates a response. Without this, a support bot hallucinates policies, leaks context between tickets, and burns through token budgets on repetitive classification tasks. This guide walks through a production-ready architecture that pairs an NLU pipeline with an LLM backend, and shows how to run it on infrastructure that does not penalize long conversations.

Architecture Overview

A reliable support agent is not a single prompt. It is a pipeline. A practical production system separates concerns into discrete stages:

Ingestion: Receive the raw user message and normalize it.
NLU: Classify intent, extract entities (order IDs, emails, product names), and tag sentiment.
Guardrails: Enforce policy checks, redact PII, and block disallowed actions.
Retrieval: Fetch relevant documentation or past tickets.
LLM: Generate a response with grounded context and available tools.
Tool Executor: Query CRMs, update tickets, or initiate refunds.
Post-processing: Format the output and log the interaction for audit.

This design keeps the LLM focused on reasoning and language generation while deterministic code handles routing and safety.

NLU Layer: Intent and Entity Extraction

Intent classification and named entity recognition are the foundation of a support agent. They determine whether a user is requesting a refund, reporting a bug, or updating an account. Extracting structured data upstream prevents the LLM from guessing order numbers or misinterpreting urgency.

Traditionally, teams fine-tune small classifiers for this. Today, you can delegate NLU to a capable LLM with a strict JSON schema. On token-based platforms, long system prompts plus few-shot examples make this expensive at scale. Because Oxlo.ai charges a flat rate per request, the cost of a verbose classification prompt with chain-of-thought reasoning is identical to a minimal one. This lets you use richer prompts and comprehensive examples without budget surprises.

import os
import json
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.getenv("OXLO_API_KEY")
)

def parse_user_input(text: str) -> dict:
    system_prompt = (
        "You are an NLU engine. Analyze the user message and return a JSON object "
        "with exactly these keys: intent, entities (list), sentiment, urgency."
    )
    response = client.chat.completions.create(
        model="qwen3-32b",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": text}
        ],
        response_format={"type": "json_object"},
        max_tokens=512
    )
    return json.loads(response.choices[0].message.content)

Memory and Context Management

Support tickets are rarely single-turn. A conversation may span dozens of messages, and the agent must remember prior authorizations, case numbers, and escalation history. Summarization helps, but sometimes you need the full transcript to resolve edge cases.

Oxlo.ai hosts models with extended context windows that are ideal for this workload. Kimi K2.6 offers 131K context for advanced reasoning and agentic coding, while DeepSeek V4 Flash supports 1M context for efficient processing of extremely long threads. Because Oxlo.ai uses request-based pricing rather than per-token metering, ingesting a full conversation history plus several knowledge base articles in one call does not inflate your bill. This removes the incentive to strip away valuable context for cost reasons.

Tool Use for Backend Integration

A support agent that only chats is an FAQ bot. Production agents must query order management systems, initiate returns, and update CRM records. Function calling lets the LLM request structured actions and receive results before formulating a final answer.

Oxlo.ai supports function calling across its chat models, so you can define tools and let the model decide when to invoke them. Below is a minimal pattern for a tool-enabled turn.

tools = [
    {
        "type": "function",
        "function": {
            "name": "get_order_status",
            "description": "Retrieve the status of a customer order by ID",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"},
                    "email": {"type": "string", "format": "email"}
                },
                "required": ["order_id"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "create_support_ticket",
            "description": "Create an internal support ticket",
            "parameters": {
                "type": "object",
                "properties": {
                    "subject": {"type": "string"},
                    "body": {"type": "string"},
                    "priority": {"type": "string", "enum": ["low", "normal", "high"]}
                },
                "required": ["subject", "body"]
            }
        }
    }
]

def run_agent_turn(history: list, user_message: str) -> str:
    history.append({"role": "user", "content": user_message})
    
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=history,
        tools=tools,
        tool_choice="auto"
    )
    
    message = response.choices[0].message
    
    if message.tool_calls:
        tool_call = message.tool_calls[0]
        result = execute_tool(tool_call)  # your business logic here
        
        history.append({
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": json.dumps(result)
        })
        
        final_response = client.chat.completions.create(
            model="llama-3.3-70b",
            messages=history
        )
        return final_response.choices[0].message.content
    
    return message.content

Building the Orchestration Layer

The orchestrator ties NLU, guardrails, and the LLM together. It routes simple requests to deterministic handlers, escalates angry users to humans, and delegates complex reasoning to the LLM. A minimal state machine works well:

Parse the incoming message with the NLU function.
If intent is refund_request and sentiment is angry, escalate immediately.
If intent is order_status and an order_id is present, inject the tool result into context.
If no tool is needed, generate a grounded response with retrieved documentation.

Because Oxlo.ai offers no cold starts on popular models, this loop stays responsive even when traffic spikes.

Why Request-Based Pricing Matters for Agents

Agentic workloads are inherently multi-turn and long-context. A single support ticket might involve classification, retrieval, one or more tool calls, and a final summarization. On token-based platforms, every extra sentence in the system prompt and every line of chat history adds marginal cost. The result is unpredictable billing that discourages thorough context gathering and rich few-shot prompting.

Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. For support agents, this means you can pass full conversation histories, detailed knowledge base context, and verbose few-shot examples without watching metered tokens accumulate. In many long-context and agentic scenarios, this can be 10-100x cheaper than token-based alternatives. See https://oxlo.ai/pricing for current plan details.

DEV Community