Building Question Answering Models with LLMs: A Step-by-Step Guide

We are going to build a retrieval-augmented question answering agent that answers questions from a private text corpus. This is useful for support teams, legal researchers, or anyone who needs grounded answers without fine-tuning. We will use a local embedding model for retrieval and an Oxlo.ai LLM for generation.

What you'll need

• Python 3.10 or newer.
• The OpenAI SDK and a few helpers: pip install openai sentence-transformers numpy.
• An Oxlo.ai API key from https://portal.oxlo.ai.

Step 1: Set up the Oxlo.ai client

Import the OpenAI SDK and point it at Oxlo.ai. I keep the base URL and key inline so the snippet is copy-paste ready.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

Step 2: Load and chunk your documents

I have hardcoded a short article so the script runs without external files. In production you would load PDFs or Markdown from disk. I split on double newlines to keep paragraphs intact.

DOCUMENT = """Apollo 11 was the spaceflight that landed the first humans on the Moon.
Commander Neil Armstrong and lunar module pilot Buzz Aldrin landed the Apollo Lunar Module Eagle on July 20, 1969.
Armstrong became the first person to step onto the lunar surface six and a half hours later on July 21.
He famously said, "That's one small step for [a] man, one giant leap for mankind."
Aldrin joined him 19 minutes later, and they spent about two and a quarter hours together exploring the site they had named Tranquility Base.
The third crew member, Michael Collins, flew the command module Columbia alone in lunar orbit while they were on the surface.
Armstrong and Aldrin collected 47.5 pounds of lunar material to bring back to Earth.
The mission was launched by a Saturn V rocket from Kennedy Space Center on Merritt Island, Florida.
"""

chunks = [chunk.strip() for chunk in DOCUMENT.split("\n\n") if chunk.strip()]

Step 3: Build a local embedding index

I use sentence-transformers running locally to encode the chunks. This keeps the retrieval step free and fast, while the heavy generation work is sent to Oxlo.ai.

from sentence_transformers import SentenceTransformer
import numpy as np

embedder = SentenceTransformer("all-MiniLM-L6-v2")
chunk_embeddings = embedder.encode(chunks, convert_to_numpy=True)

Step 4: Retrieve relevant context

When a question comes in, I embed it with the same model and compute cosine similarity against the index. I return the top three chunks to keep the context window focused.

def retrieve(query: str, top_k: int = 3):
    query_embedding = embedder.encode([query], convert_to_numpy=True)
    similarities = np.dot(chunk_embeddings, query_embedding.T).squeeze()
    top_indices = np.argsort(similarities)[-top_k:][::-1]
    return [chunks[i] for i in top_indices]

Step 5: Define the QA system prompt

The system prompt constrains the model to the retrieved evidence and prevents hallucination. I also force brevity so the answers stay scannable.

SYSTEM_PROMPT = (
    "You are a strict question-answering assistant. "
    "Answer using ONLY the context provided below. "
    "If the answer is not in the context, reply 'I don't know.' "
    "Keep your answer concise and factual."
)

Step 6: Generate answers with Oxlo.ai

Now I wire retrieval and generation together. I join the top chunks into a single context block, pass it to the model along with the question, and return the generated answer. Because Oxlo.ai uses flat per-request pricing, you can pass the full retrieved context on every call without worrying about token length driving up cost.

def answer_question(question: str) -> str:
    context = "\n\n".join(retrieve(question))
    user_message = f"Context:\n{context}\n\nQuestion: {question}"

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content

Run it

Call the function with a question that is answerable from the text. If you want to experiment with reasoning models, swap llama-3.3-70b for deepseek-v3.2 or qwen-3-32b without changing any other code.

if __name__ == "__main__":
    q = "What did Neil Armstrong say when he stepped onto the Moon?"
    print(f"Q: {q}")
    print(f"A: {answer_question(q)}")

Expected output:

Q: What did Neil Armstrong say when he stepped onto the Moon?
A: Neil Armstrong said, "That's one small step for [a] man, one giant leap for mankind."

Next steps

Swap the hardcoded string for a real document loader such as PyPDF or Unstructured so you can point the agent at PDFs or Markdown folders. If you need multi-turn conversations, append each exchange to the messages list and let users ask follow-up questions about the same corpus.