Building a Question Answering System using LLM: A Step-by-Step Guide

We are going to build a retrieval-augmented question answering system that grounds LLM responses in a private text corpus. This is the same pattern used for internal help desks, legal document search, and developer documentation bots. Because Oxlo.ai charges a flat rate per request instead of per token, you can stuff large retrieved contexts into the prompt without worrying about ballooning costs.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK and NumPy: pip install openai numpy

Step 1: Initialize the Oxlo.ai client

Oxlo.ai exposes an OpenAI-compatible API, so the only changes are the base URL and the model identifier. I will use llama-3.3-70b for generation because it handles instruction following reliably.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Create a document store

For this tutorial we will use a hardcoded list of support articles. In production you would load these from a database or a scraped documentation folder.

documents = [
    {
        "id": "doc-1",
        "text": "Oxlo.ai uses flat per-request pricing. Each API call costs one fixed amount regardless of prompt length, which makes long-context workloads significantly cheaper than token-based billing."
    },
    {
        "id": "doc-2",
        "text": "The Free plan includes 60 requests per day and access to 16 models. The Pro plan offers 1,000 requests per day and access to all 45 models."
    },
    {
        "id": "doc-3",
        "text": "Oxlo.ai supports streaming responses, function calling, JSON mode, and vision inputs. The API is fully compatible with the OpenAI SDK."
    },
    {
        "id": "doc-4",
        "text": "Popular models on Oxlo.ai include Llama 3.3 70B for general tasks, DeepSeek R1 671B MoE for deep reasoning, and Kimi K2.6 for agentic coding with vision."
    },
]

Step 3: Embed documents for retrieval

We generate embeddings with Oxlo.ai's BGE-Large model and keep them in a NumPy array. This is our bare-bones semantic search layer.

import numpy as np

def get_embedding(text):
    response = client.embeddings.create(
        model="bge-large",
        input=text,
    )
    return np.array(response.data[0].embedding)

doc_embeddings = np.array([get_embedding(d["text"]) for d in documents])

Step 4: Implement similarity search

At query time we compute cosine similarity between the question embedding and every document embedding, then return the top three chunks.

def retrieve(query, top_k=3):
    q_emb = get_embedding(query)
    norms = np.linalg.norm(doc_embeddings, axis=1) * np.linalg.norm(q_emb)
    similarities = (doc_embeddings @ q_emb) / norms
    top_indices = np.argsort(similarities)[::-1][:top_k]
    return [documents[i] for i in top_indices]

Step 5: Write the system prompt

The prompt acts as a guardrail. It forces the model to stay grounded in the retrieved context and to refuse hallucination when the answer is not present.

SYSTEM_PROMPT = """You are a precise technical support assistant.
Answer the user's question using ONLY the context provided below.
If the context does not contain the answer, say "I don't have that information."
Do not make up facts.

Context:
{context}
"""

Step 6: Generate the answer

We pull the relevant chunks, format them into the system prompt, and call the chat endpoint. Because Oxlo.ai pricing is per request, the total cost is the same whether we send one sentence or the full set of retrieved articles.

def answer_question(query):
    retrieved_docs = retrieve(query)
    context = "\n\n".join([f"[{d['id']}] {d['text']}" for d in retrieved_docs])
    filled_prompt = SYSTEM_PROMPT.format(context=context)

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": filled_prompt},
            {"role": "user", "content": query},
        ],
        temperature=0.1,
    )
    return response.choices[0].message.content, retrieved_docs

Run it

Here is a short script that asks two questions and prints the generated answer along with the source IDs so you can verify the grounding.

if __name__ == "__main__":
    questions = [
        "How does Oxlo.ai pricing work?",
        "Which model should I use for coding?",
    ]

    for q in questions:
        print(f"Q: {q}")
        ans, sources = answer_question(q)
        print(f"A: {ans}")
        print("Sources:")
        for s in sources:
            print(f"  - {s['id']}")
        print()

Example output:

Q: How does Oxlo.ai pricing work?
A: Oxlo.ai uses flat per-request pricing. Each API call costs one fixed amount regardless of prompt length, which makes long-context workloads significantly cheaper than token-based billing.
Sources:
  - doc-1

Q: Which model should I use for coding?
A: For coding, you can use DeepSeek R1 671B MoE for deep reasoning, or Kimi K2.6 for agentic coding with vision.
Sources:
  - doc-4

Next steps

Replace the in-memory document list with a persistent vector database such as Qdrant or pgvector so the system scales past a few dozen chunks. You can also add a re-ranking step or a second LLM call to compress long contexts before the final answer generation. Because Oxlo.ai uses flat per-request pricing, those extra calls stay predictable even when the context window grows. For details on plans and costs, see https://oxlo.ai/pricing.