Customer service is one of the first production environments where large language models (LLMs) have moved from experiment to critical infrastructure. Modern support pipelines now rely on LLMs for intent classification, response generation, and agentic tool use across chat, email, and voice transcripts. Unlike rigid rule-based systems, LLMs adapt to ambiguous phrasing, multilingual queries, and unstructured knowledge bases. Deploying them at scale, however, introduces architectural and economic constraints that platform choices directly determine.
From Classification to Generation
Early customer service automation used intent classifiers and slot-filling parsers to route tickets. These systems required curated training data and broke when users deviated from expected phrasing. LLMs collapsed these layers into a single reasoning step. A model can now parse a complaint, retrieve policy context, draft a resolution, and decide whether to escalate, all within one multi-turn conversation.
This shift demands more than raw parameter count. Production support workloads require function calling to query CRMs, JSON mode to emit structured ticket updates, and long context windows to ingest entire conversation histories or knowledge base articles in a single request. The infrastructure must expose these features through a stable, low-latency API without cold starts that add friction to real-time chat.
Architecture Patterns for Production
Most production LLM support stacks follow a retrieval-augmented generation (RAG) pattern. Embeddings models, such as BGE-Large or E5-Large on Oxlo.ai, index help-center articles and past resolutions. At inference time, the system retrieves relevant chunks and injects them into the prompt. For actions that require real-time data, such as checking an order status or initiating a refund, function calling lets the model invoke external APIs rather than hallucinating facts.
State management is equally important. A support session may span multiple turns, include image uploads (screenshots of errors), and require tool results to be fed back into the model. An endpoint that supports multi-turn conversations, vision input, and streaming responses keeps latency perceptible to the user while the backend orchestrates tool loops.
The Cost Structure of Context
The hidden cost in customer service LLMs is context length. A single support ticket can include a long email thread, a knowledge base article, and prior chat history. Under token-based billing, every additional paragraph increases cost linearly. For agentic workflows that iterate over tool results and append them to context, token counts compound quickly.
Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. For long-context and agentic support workloads, this can be 10-100x cheaper than token-based alternatives. Because cost does not scale with input length, teams can pass full conversation histories and large retrieved documents without trimming context to save money. See the Oxlo.ai pricing page for plan details.
Building a Support Agent with Oxlo.ai
Oxlo.ai is fully OpenAI SDK compatible, so migrating or prototyping a support agent requires only a base URL change. The platform offers function calling, JSON mode, and streaming across its chat models, with no cold starts on popular endpoints.
The following Python example shows a simple agent that can answer questions using a simulated order database. It uses Qwen 3 32B, a strong choice for multilingual reasoning and agent workflows, but you can substitute Llama 3.3 70B or another chat model from the Oxlo.ai catalog.
import openai
import json
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
def get_order_status(order_id: str):
# Simulated CRM lookup
return {"order_id": order_id, "status": "shipped", "eta": "2 days"}
tools = [
{
"type": "function",
"function": {
"name": "get_order_status",
"description": "Retrieve the current status of a customer order",
"parameters":
Top comments (0)