The Role of LLM in Chatbots and Virtual Assistants

#learnai #oxlo #ai

In this tutorial I will build a grounded customer support virtual assistant that answers refund and account questions using a company knowledge base. The goal is to give Tier-1 support teams a working CLI agent that maintains context and cites policy without racking up token-based costs. I run it on Oxlo.ai because flat per-request pricing makes long context windows cheap and predictable.

What you'll need

Python 3.10 or newer.
The OpenAI SDK: pip install openai.
An Oxlo.ai API key from https://portal.oxlo.ai.

Step 1: Scope the assistant and lock in the system prompt

A virtual assistant needs boundaries. I set the tone, persona, and escalation rules in a single system prompt so the model stays consistent across turns.

SYSTEM_PROMPT = """You are a Tier-1 support agent for Acme SaaS. Your job is to answer questions about refunds, account settings, and billing using only the knowledge base provided in your context. Be concise, polite, and accurate. If a user asks something outside refunds, billing, or account settings, say: "I do not have information on that topic. I can connect you with a human agent." Do not make up policies."""

Step 2: Initialize the Oxlo.ai client and test a single turn

I point the OpenAI SDK at Oxlo.ai and send one message to confirm the endpoint and credentials are live. I use Llama 3.3 70B as the general-purpose flagship.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

user_message = "How do I request a refund?"

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": user_message},
    ],
)

print(response.choices[0].message.content)

Step 3: Add conversation memory

Real assistants remember what the user just said. I keep a running messages list so the model sees the full thread on every call.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
]

while True:
    user_input = input("User: ")
    if user_input.lower() in {"exit", "quit"}:
        break

    messages.append({"role": "user", "content": user_input})

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages,
    )

    assistant_reply = response.choices[0].message.content
    print(f"Assistant: {assistant_reply}")
    messages.append({"role": "assistant", "content": assistant_reply})

Step 4: Ground answers with a knowledge base article

Hallucinated refund policies are dangerous. I prepend a full FAQ to the system prompt so every answer is policy-backed. Because Oxlo.ai charges per request rather than per token, stuffing a few thousand tokens of context into every call does not change the price. That predictability matters when you scale to hundreds of conversations.

KNOWLEDGE_BASE = """
Acme SaaS Refund and Account Policy FAQ

Q: How do I request a refund?
A: Email support@acme.example with your invoice number within 14 days of purchase. Refunds are processed in 5 to 7 business days.

Q: Can I change my plan mid-cycle?
A: Yes. Upgrades take effect immediately and are prorated. Downgrades take effect at the next billing cycle.

Q: What payment methods are accepted?
A: Credit cards and ACH transfers. We do not accept cryptocurrency.

Q: How do I reset my password?
A: Click "Forgot password" on the login page. Reset links expire after 1 hour.
"""

GROUNDED_PROMPT = SYSTEM_PROMPT + "\n\nUse the following knowledge base to answer questions:\n" + KNOWLEDGE_BASE

Step 5: Wire everything into a runnable CLI

Here is the complete script that combines the prompt, memory, and knowledge base into a single runnable CLI.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

SYSTEM_PROMPT = """You are a Tier-1 support agent for Acme SaaS. Your job is to answer questions about refunds, account settings, and billing using only the knowledge base provided in your context. Be concise, polite, and accurate. If a user asks something outside refunds, billing, or account settings, say: "I do not have information on that topic. I can connect you with a human agent." Do not make up policies."""

KNOWLEDGE_BASE = """
Acme SaaS Refund and Account Policy FAQ

Q: How do I request a refund?
A: Email support@acme.example with your invoice number within 14 days of purchase. Refunds are processed in 5 to 7 business days.

Q: Can I change my plan mid-cycle?
A: Yes. Upgrades take effect immediately and are prorated. Downgrades take effect at the next billing cycle.

Q: What payment methods are accepted?
A: Credit cards and ACH transfers. We do not accept cryptocurrency.

Q: How do I reset my password?
A: Click "Forgot password" on the login page. Reset links expire after 1 hour.
"""

messages = [
    {"role": "system", "content": SYSTEM_PROMPT + "\n\nUse the following knowledge base to answer questions:\n" + KNOWLEDGE_BASE},
]

print("Support agent is running. Type 'exit' to quit.")

while True:
    user_input = input("User: ")
    if user_input.lower() in {"exit", "quit"}:
        break

    messages.append({"role": "user", "content": user_input})

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=messages,
    )

    assistant_reply = response.choices[0].message.content
    print(f"Assistant: {assistant_reply}")
    messages.append({"role": "assistant", "content": assistant_reply})

Run it

Save the file as support_agent.py, export your key, and run it. Below is a sample transcript from my session.

export OXLO_API_KEY="sk-oxlo.ai-..."
python support_agent.py

Support agent is running. Type 'exit' to quit.
User: How do I request a refund?
Assistant: Email support@acme.example with your invoice number within 14 days of purchase. Refunds are processed in 5 to 7 business days.

User: Can I downgrade today?
Assistant: Downgrades take effect at your next billing cycle, not immediately.

User: Do you have a status page?
Assistant: I do not have information on that topic. I can connect you with a human agent.

Wrap-up and next steps

This assistant is already useful for internal testing, but two upgrades make it production ready. First, swap the static FAQ for dynamic retrieval with Oxlo.ai's embeddings endpoint and BGE-Large, then keep the per-request savings. Second, wrap the loop in FastAPI and stream responses back to a frontend using Oxlo.ai's streaming support.