shashank ms

Posted on Sep 19

Creating Chatbots with LLM and RAG: A Technical Guide

#engineering #oxlo #ai

We are building an internal support chatbot that answers product questions by retrieving snippets from a Markdown knowledge base and generating responses with an LLM hosted on Oxlo.ai. This pattern, retrieval-augmented generation, cuts hallucinations and keeps answers current without fine-tuning. I will walk through a minimal but production-ready version I actually deploy for my own team's wiki.

What you'll need

Python 3.10 or newer
pip install openai numpy
An Oxlo.ai API key from https://portal.oxlo.ai. Oxlo.ai's request-based pricing makes this cheap to experiment with, even when sending long context chunks on every turn. See https://oxlo.ai/pricing for details.
A few Markdown files, or you can use the inline sample docs I provide below.

1. Initialize the Oxlo.ai client and load the knowledge base

I start by pointing the OpenAI SDK at Oxlo.ai and defining a tiny knowledge base of help articles. Keeping the docs inline means you can run this immediately without hunting for files.

from openai import OpenAI
import numpy as np

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

DOCS = [
    {
        "title": "Resetting API Keys",
        "content": "To reset your API key, open the Dashboard, navigate to Settings, and click 'Regenerate Key'. The old key expires in 24 hours."
    },
    {
        "title": "Uptime SLA",
        "content": "Our paid clusters guarantee 99.99% uptime, measured monthly. If we fall below this, contact support for a service credit."
    },
    {
        "title": "SSO Setup",
        "content": "SSO is available on Enterprise plans. Go to Settings, then Security, and paste your SAML 2.0 metadata URL. Test the connection before enforcing it for all users."
    },
    {
        "title": "Rate Limits",
        "content": "Free tiers are limited to 60 requests per day. Pro tiers allow 1,000 requests per day. Burst limits are 100 requests per minute."
    },
]

2. Chunk the documents

I split each article into overlapping chunks so the retriever can target specific sentences instead of stuffing entire articles into the prompt.

def chunk_text(text, chunk_size=40, overlap=10):
    words = text.split()
    chunks = []
    for i in range(0, len(words), chunk_size - overlap):
        chunk = words[i:i + chunk_size]
        chunks.append(" ".join(chunk))
        if i + chunk_size >= len(words):
            break
    return chunks

chunks = []
for doc in DOCS:
    for c in chunk_text(doc["content"]):
        chunks.append({"title": doc["title"], "text": c})

print(f"Generated {len(chunks)} chunks")

3. Embed chunks with Oxlo.ai

I embed every chunk using Oxlo.ai's BGE-Large model and cache the vectors in a NumPy array. Because Oxlo.ai charges per request, not per token, embedding these small chunks costs the same whether they are ten words or a thousand.

def get_embeddings(texts, model="bge-large"):
    texts = [t.replace("\n", " ") for t in texts]
    response = client.embeddings.create(model=model, input=texts)
    return [item.embedding for item in response.data]

chunk_texts = [c["text"] for c in chunks]
chunk_vectors = np.array(get_embeddings(chunk_texts))
print(f"Embeddings shape: {chunk_vectors.shape}")

4. Build the cosine-similarity retriever

At query time I embed the question the same way, then score it against every chunk and return the top three matches.

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def retrieve(query, top_k=3):
    q_vec = np.array(get_embeddings([query])[0])
    scores = [cosine_similarity(q_vec, c_vec) for c_vec in chunk_vectors]
    top_idx = np.argsort(scores)[-top_k:][::-1]
    return [chunks[i] for i in top_idx]

# Quick sanity check
for hit in retrieve("How do I rotate my API key?"):
    print(f"- {hit['title']}: {hit['text']}")

5. Write the system prompt

The system prompt is the only guardrail that matters in this stack. I tell the model to stay grounded in the retrieved context and to admit ignorance.

SYSTEM_PROMPT = """You are a terse support engineer. Answer the user's question using ONLY the context provided below.
If the context does not contain the answer, say "I don't have that information."
Cite the article title in parentheses after each fact.

Context:
{context}
"""

6. Generate answers with Llama 3.3 70B on Oxlo.ai

I assemble the retrieved snippets into the context string, inject them into the system prompt, and call Oxlo.ai. I use Llama 3.3 70B because it follows system instructions reliably and handles long context well.

def answer_question(user_message):
    hits = retrieve(user_message)
    context_blocks = [f"[{h['title']}] {h['text']}" for h in hits]
    context = "\n\n".join(context_blocks)
    system_prompt = SYSTEM_PROMPT.format(context=context)

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content, hits

7. Add a simple chat loop

I wrap the retriever and generator in a small CLI so I can test multi-turn questions without restarting the interpreter.

if __name__ == "__main__":
    print("Support bot ready. Type 'exit' to quit.\n")
    while True:
        user_input = input("User: ").strip()
        if user_input.lower() in {"exit", "quit"}:
            break
        if not user_input:
            continue
        reply, sources = answer_question(user_input)
        print(f"Bot: {reply}\n")
        print("Sources:")
        for s in sources:
            print(f"  - {s['title']}")
        print()

Run it

Save the script as support_bot.py, export your key, and run it:

export OXLO_API_KEY="sk-oxlo.ai-..."
python support_bot.py

Example session:

Support bot ready. Type 'exit' to quit.

User: How do I reset my API key?
Bot: Open the Dashboard, navigate to Settings, and click 'Regenerate Key'. The old key expires in 24 hours. (Resetting API Keys)

Sources:
  - Resetting API Keys

User: What is the uptime guarantee?
Bot: Our paid clusters guarantee 99.99% uptime, measured monthly. If we fall below this, contact support for a service credit. (Uptime SLA)

Sources:
  - Uptime SLA

User: Do you support OAuth2?
Bot: I don't have that information.

Sources:
  - SSO Setup
  - Resetting API Keys
  - Uptime SLA

Wrap-up

This stack is easy to extend. Swap the in-memory NumPy store for pgvector or Chroma if you need persistence across restarts, or wrap the answer_question function in a FastAPI endpoint and wire it to Slack. If you want better reasoning for complex multi-hop questions, try switching the chat model to kimi-k2.6 or deepseek-v3.2 on Oxlo.ai without changing any other code.

DEV Community