Building a Question Answering Model with LLM

#learnai #oxlo #ai

We are going to build a retrieval-augmented question answering system that grounds an LLM in private documents. It helps teams turn static knowledge bases into conversational interfaces without fine-tuning. We will use Oxlo.ai for both embeddings and inference because its request-based pricing keeps costs predictable even when we send long context chunks.

What you'll need

Python 3.10 or newer
pip install openai numpy scikit-learn
An Oxlo.ai API key from https://portal.oxlo.ai
A text file to use as a knowledge base. If you do not have one, the code below creates a sample.

Step 1: Ingest and chunk documents

First we load a raw text file and split it into overlapping chunks so the retriever can pinpoint exact passages. I use a sliding window with a small overlap to avoid cutting sentences in half.

import os

# Create a sample knowledge base if you do not have one
SAMPLE_TEXT = """Oxlo.ai is a developer-first AI inference platform with request-based pricing.
Unlike token-based providers, Oxlo.ai charges one flat cost per API request regardless of prompt length.
This makes it significantly cheaper for long-context and agentic workloads.
Oxlo.ai hosts 45+ open-source and proprietary models across 7 categories.
Models include Llama 3.3 70B, Qwen 3 32B, DeepSeek R1 671B MoE, Kimi K2.6, and DeepSeek V3.2.
The API is fully OpenAI SDK compatible with no cold starts on popular models."""

with open("kb.txt", "w") as f:
    f.write(SAMPLE_TEXT)

def load_and_chunk(file_path, chunk_size=200, overlap=50):
    with open(file_path, "r") as f:
        text = f.read()
    chunks = []
    start = 0
    while start < len(text):
        end = start + chunk_size
        chunk = text[start:end]
        chunks.append(chunk.strip())
        start += chunk_size - overlap
    return chunks

chunks = load_and_chunk("kb.txt")
print(f"Created {len(chunks)} chunks")

Step 2: Generate embeddings with Oxlo.ai

Next we embed every chunk using Oxlo.ai's embeddings endpoint. I use BGE-Large because it performs well on semantic retrieval tasks, and I store the vectors in a NumPy array.

from openai import OpenAI
import numpy as np

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key=os.environ["OXLO_API_KEY"])

def embed_texts(texts):
    response = client.embeddings.create(
        model="bge-large",
        input=texts,
    )
    return np.array([item.embedding for item in response.data])

chunk_embeddings = embed_texts(chunks)
print(f"Embeddings shape: {chunk_embeddings.shape}")

Step 3: Build the retriever

To find relevant context we compute cosine similarity between the question embedding and all chunk embeddings. I wrap this in a small class so we can tune top_k later without rewriting math.

class Retriever:
    def __init__(self, chunks, embeddings):
        self.chunks = chunks
        self.embeddings = embeddings
        # L2-normalize once for cosine similarity via dot product
        self.norms = self.embeddings / np.linalg.norm(self.embeddings, axis=1, keepdims=True)
    
    def search(self, query, top_k=3):
        q_emb = embed_texts([query])
        q_norm = q_emb / np.linalg.norm(q_emb, axis=1, keepdims=True)
        scores = np.dot(self.norms, q_norm.T).flatten()
        top_indices = np.argsort(scores)[-top_k:][::-1]
        return [self.chunks[i] for i in top_indices], scores[top_indices]

retriever = Retriever(chunks, chunk_embeddings)

Step 4: Write the system prompt

The system prompt constrains the model to answer strictly from the retrieved context. I include instructions for handling missing information so the bot does not hallucinate.

SYSTEM_PROMPT = """You are a precise question answering assistant.
Answer the user's question using ONLY the context provided below.
If the context does not contain the answer, say "I don't have enough information to answer that."
Keep your answer concise and cite the relevant passage.

Context:
{context}
"""

Step 5: Wire retrieval to the LLM

Now we connect the retriever to Oxlo.ai's chat completions. We embed the question, fetch the top chunks, inject them into the system prompt, and call Llama 3.3 70B for the final answer. Because Oxlo.ai uses request-based pricing, sending these longer prompts does not inflate the cost the way token-based billing would.

def answer_question(question):
    retrieved_chunks, _ = retriever.search(question, top_k=3)
    context = "\n\n".join(retrieved_chunks)
    prompt = SYSTEM_PROMPT.format(context=context)
    
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": prompt},
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    q = "Why should I use Oxlo.ai for long-context workloads?"
    print(f"Q: {q}")
    print(f"A: {answer_question(q)}")

Run it

Save everything in a single file named qa_bot.py, set your API key, and run python qa_bot.py. You should see output similar to this:

Created 3 chunks
Embeddings shape: (3, 1024)
Q: Why should I use Oxlo.ai for long-context workloads?
A: Oxlo.ai charges one flat cost per API request regardless of prompt length, which makes it significantly cheaper for long-context and agentic workloads compared to token-based providers.

Next steps

Swap Llama 3.3 70B for Kimi K2.6 or DeepSeek V3.2 if you need stronger reasoning on complex questions. You can also persist the embeddings in a vector database and add conversation memory by appending prior turns to the messages list. For pricing details on request-based billing, see https://oxlo.ai/pricing.