Building a Chatbot with LLM and RAG: A Step-by-Step Guide

#aiinfrastructure #oxlo #ai

Retrieval-Augmented Generation (RAG) is the standard pattern for grounding LLM chatbots in private or domain-specific data. Instead of relying solely on parametric knowledge, a RAG pipeline retrieves relevant text chunks at query time and injects them into the prompt. The result is a chatbot that answers accurately about your documents without requiring fine-tuning. In this guide, you will build a complete RAG chatbot using Python, vector search, and an LLM inference API. We will use Oxlo.ai for generation because its request-based pricing removes the cost penalty typically associated with long retrieved contexts.

Architecture Overview

A minimal RAG chatbot has four components: a document chunker, an embedding model, a vector store, and a chat model. The flow is straightforward. Documents are split into chunks, converted to vectors, and indexed. When a user asks a question, the system embeds the query, retrieves the top-k chunks, and concatenates them into a system or user prompt. The LLM then generates an answer conditioned on that retrieved evidence.

Project Setup and Dependencies

You need Python 3.10+, the OpenAI Python SDK, and a vector store such as Chroma or FAISS. Oxlo.ai is fully OpenAI SDK compatible, so you can instantiate a client with Oxlo.ai's base URL and your API key.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

Install the remaining dependencies locally.

pip install openai chromadb numpy

Step 1: Document Ingestion and Chunking

For this example, assume you have a list of raw text strings. In production these might come from PDFs, Markdown files, or a web crawl. The goal is to produce small, semantically coherent chunks.

def chunk_documents(documents, chunk_size=512, overlap=50):
    chunks = []
    for doc in documents:
        for i in range(0, len(doc), chunk_size - overlap):
            chunks.append(doc[i:i + chunk_size])
    return chunks

documents = [
    "Oxlo.ai is a developer-first AI inference platform...",
    "Request-based pricing means one flat cost per API request..."
]
chunks = chunk_documents(documents)

Step 2: Generating Embeddings

Once your documents are chunked, you need to vectorize them. While you can run an embedding model locally, Oxlo.ai exposes BGE-Large and E5-Large through the same OpenAI-compatible client. This keeps your pipeline unified and simplifies billing.

def get_embedding(text: str) -> list[float]:
    response = client.embeddings.create(
        model="bge-large",
        input=text
    )
    return response.data[0].embedding

embeddings = [get_embedding(c) for c in chunks]

Step 3: Retrieval Logic

With the index built, retrieval is a similarity search. The following example uses ChromaDB to store vectors and query them at runtime.

import chromadb

chroma_client = chromadb.Client()
collection = chroma_client.create_collection(name="rag_docs")

collection.add(
    ids=[f"chunk_{i}" for i in range(len(chunks))],
    embeddings=embeddings,
    documents=chunks
)

def retrieve(query: str, top_k: int = 3):
    q_embed = get_embedding(query)
    results = collection.query(query_embeddings=[q_embed], n_results=top_k)
    return results["documents"][0]

Step 4: Chat Loop with LLM Inference

The final step is sending the retrieved context to the LLM. Because RAG injections can make prompts long, token-based

DEV Community