Integrating LLM with Existing Chatbots: A Step-by-Step Guide

Most enterprise chatbots still run on intent classifiers and rigid decision trees. They handle the happy path well, but break on ambiguity, edge cases, and multi-turn context. Instead of rebuilding from scratch, the pragmatic approach is to wire a modern LLM into your existing pipeline as a fallback, enricher, or full handoff layer. This guide walks through the architectural patterns, code patterns, and cost considerations for doing exactly that, using Oxlo.ai as the inference backend.

Why Integrate Instead of Rebuild

Existing platforms have validated business logic, integrations, and compliance boundaries. A rip-and-replace introduces risk. An integration strategy lets you preserve your current NLU investment while using an LLM for out-of-scope queries, personalized rewriting, or complex agentic tasks. You improve coverage incrementally without retraining your entire stack.

Integration Patterns

There are three practical ways to add an LLM to a legacy bot without disrupting the user experience.

Pattern A: LLM as Fallback Router. When your classifier confidence drops below a threshold, route the utterance to an LLM. The LLM can either answer directly or map the request to a known intent and return structured JSON.

Pattern B: Response Enricher. Keep your existing response generation, but pass the draft through an LLM to adjust tone, translate, or expand bullets into natural language.

Pattern C: Agentic Handoff. For workflows that require multi-step reasoning or tool use, hand the session to an LLM with function calling enabled. Once the task completes, return control to the legacy bot.

Prerequisites and Tooling

• A webhook or message broker for your current chatbot.
• An Oxlo.ai API key. Sign up at Oxlo.ai to choose a plan.
• The OpenAI SDK (Python or Node.js). Oxlo.ai exposes https://api.oxlo.ai/v1, so existing OpenAI client code works with a one-line base URL change.

Step 1: Audit Your Current Bot Pipeline

Map your current confidence scores and fallback rates. Identify the top 20% of unrecognized utterances that generate the most support tickets. Those are your first candidates for LLM augmentation.

Step 2: Set Up the Oxlo.ai Client

Because Oxlo.ai is fully OpenAI SDK compatible, you do not need to refactor your client initialization logic beyond swapping the base URL and key.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

Step 3: Implement a Fallback Gateway

Use your legacy classifier as the primary path, and invoke Oxlo.ai only when confidence is low. Here we use Llama 3.3 70B for general-purpose fallback reasoning, but you could select Qwen 3 32B if the workflow is multilingual or agentic.

import json

def handle_user_message(user_text, conversation_history, intent_confidence):
    if intent_confidence < 0.7:
        response = client.chat.completions.create(
            model="llama-3.3-70b",
            messages=[
                {
                    "role": "system",
                    "content": (
                        "You are a helpful assistant for Acme Corp. "
                        "Answer concisely. If you cannot help, return JSON "
                        "with action: 'escalate'."
                    )
                },
                *conversation_history,
                {"role": "user", "content": user_text}
            ],
            response_format={"type": "json_object"}
        )
        return json.loads(response.choices[0].message.content)
    else:
        return legacy_bot_response(user_text)

Step 4: Inject Conversation Context

LLMs need prior turns to maintain coherence. Format your existing chat logs into the OpenAI messages schema.

def format_history(internal_turns):
    messages = []
    for turn in internal_turns:
        role = "user" if turn["speaker"] == "customer" else "assistant"
        messages.append({"role": role, "content": turn["text"]})
    return messages

Because Oxlo.ai uses request-based pricing, you can include full conversation history in every request without watching token meters accumulate. This is especially useful for long support threads.

Step 5: Handle Tool Use and Escalation

If the LLM needs to check an order status or book a meeting, use function calling. Function calling is supported across Oxlo.ai's chat models, so your LLM can query internal APIs before formulating a final answer.

tools = [
    {
        "type": "function",
        "function": {
            "name": "check_order_status",
            "description": "Retrieve status for an order",
            "parameters": {
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"}
                },
                "required": ["order_id"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="qwen-3-32b",
    messages=messages,
    tools=tools,
    tool_choice="auto"
)

if response.choices[0].message.tool_calls:
    # Execute the tool, append the result, and call the model again
    ...

Step 6: Streaming Responses

To keep the chat interface responsive, stream tokens directly to the client.

stream = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=messages,
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        yield chunk.choices[0].delta.content

Oxlo.ai supports streaming, so you can pipe tokens straight into your chat widget with no perceptible cold start on popular models.

Why Request-Based Pricing Fits Chatbot Workloads

Chatbot sessions naturally accumulate context. System prompts, conversation history, and retrieval documents can quickly inflate prompt token counts. With token-based providers, costs scale linearly with input length, which makes long-context and agentic workloads expensive and unpredictable.

Oxlo.ai charges one flat cost per API request regardless of prompt length, so sending a full conversation thread costs the same as sending a single word. For high-context workloads, request-based pricing can be 10 to 100 times cheaper than token-based alternatives. See https://oxlo.ai/pricing for current plan details.

Production Checklist

• Timeouts. Set aggressive HTTP timeouts and circuit breakers around the LLM fallback path so a slow inference call does not hang your primary bot.
• Context windows. Implement truncation or summarization for sessions that approach the model's context limit.
• PII filtering. Strip sensitive fields before sending to any third-party inference provider.
• Schema validation. Validate JSON schema client-side even when using JSON mode.
• Model selection. Choose Qwen 3 32B for multilingual agent workflows, Llama 3.3 70B for general fallback, or DeepSeek R1 671B MoE for deep reasoning tasks.

Conclusion

Integrating an LLM into an existing chatbot is a low-risk, high-return migration path. You keep your validated business logic and gain modern generative capabilities exactly where you need them. Oxlo.ai makes this integration straightforward: the OpenAI SDK compatibility means your fallback gateway requires only a base URL change, the request-based pricing model rewards you for maintaining full context, and the broad model catalog lets you tune latency and capability to each use case. Start with a fallback router, measure resolution rates, and expand from there.