Building Chatbots with LLM and RAG: A Step-by-Step Guide

#aiinfrastructure #oxlo #ai

Retrieval-Augmented Generation (RAG) is the standard pattern for grounding LLM chatbots in private data. Instead of relying solely on parametric knowledge, a RAG pipeline retrieves relevant document chunks at query time and injects them into the LLM context window. This gives you factual, source-cited responses without retraining models. In this guide, you will build a complete RAG chatbot using Python, vector search, and the Oxlo.ai inference API.

Architecture Overview

A production RAG chatbot has four stages. Ingestion loads raw documents and splits them into chunks. Embedding converts each chunk into a dense vector. Retrieval searches those vectors for the closest matches to a user question. Generation feeds the retrieved chunks into an LLM prompt to synthesize an answer. Each stage has implementation choices that affect latency, cost, and accuracy.

Environment Setup

You need Python 3.10+ and a few packages. Because Oxlo.ai is fully OpenAI SDK compatible, you can use the official openai Python client with only a base URL change.

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.getenv("OXLO_API_KEY"),
    base_url="https://api.oxlo.ai/v1"
)

Install the remaining dependencies:

pip install openai chromadb langchain-text-splitters pypdf

Step 1: Document Ingestion and Chunking

Long documents must be split into semantically coherent chunks. A chunk size of 512 to 1024 tokens with overlap usually works well for technical documentation. The example below loads a PDF and splits it with recursive character splitting.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from pypdf import PdfReader

reader = PdfReader("documentation.pdf")
text = "\n\n".join([page.extract_text() or "" for page in reader.pages])

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,
    chunk_overlap=64,
    separators=["\n\n", "\n", ". ", " ", ""]
)
chunks = splitter.split_text(text)

Step 2: Embedding with Oxlo.ai

Convert chunks to vectors using an embeddings model. Oxlo.ai offers BGE-Large and E5-Large through a fully compatible embeddings endpoint. Because Oxlo.ai uses request-based pricing rather than token-based billing, embedding a large batch of chunks in a single API call costs the same flat amount regardless of total token count. For current plan details, see https://oxlo.ai/pricing.

def get_embeddings(texts):
    response = client.embeddings.create(
        model="bge-large",
        input=texts
    )
    return [item.embedding for item in response.data]

# Batch embed in groups of 100 to stay within payload limits
batch_size = 100
all_embeddings = []
for i in range(0, len(chunks), batch_size):
    batch = chunks[i:i + batch_size]
    all_embeddings.extend(get_embeddings(batch))

Step 3: Vector Storage

Store the vectors and their text in ChromaDB so you can query them at runtime. Chroma runs in-memory for prototyping and can be persisted to disk.

import chromadb

chroma_client = chromadb.Client()
collection = chroma_client.create_collection(name="docs")

collection.add(
    documents=chunks,
    embeddings=all_embeddings,
    ids=[f"chunk_{i}" for i in range(len(chunks))]
)

DEV Community