Building Question Answering Models with LLM

We are going to build a retrieval-augmented question answering agent that ingests a small knowledge base and returns cited answers. This pattern is useful for internal support bots, legal document review, or any workflow where hallucinated facts are unacceptable. I will keep the retrieval layer minimal so the focus stays on the LLM integration and prompt structure.

What you'll need

• Python 3.10 or newer.
• The OpenAI SDK installed with pip install openai.
• An Oxlo.ai API key from https://portal.oxlo.ai. Review the flat per-request pricing at https://oxlo.ai/pricing. Because Oxlo.ai does not bill by the token, stuffing large contexts into a single request does not inflate cost the way token-based providers do.

1. Set up the Oxlo.ai client

Oxlo.ai is fully OpenAI SDK compatible, so the setup is a single line change of base URL and key. I instantiate the client once and reuse it across requests.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

2. Prepare a toy knowledge base

I will use a plain Python list of documents. Each entry has an ID and a text chunk. In production you would load this from chunked PDFs, HTML, or a Notion export, but the structure stays the same.

DOCUMENTS = [
    {
        "id": "doc-001",
        "text": "Oxlo.ai provides flat per-request pricing for open-source LLMs. Unlike token-based providers, the cost does not scale with prompt length, making it significantly cheaper for long-context workloads."
    },
    {
        "id": "doc-002",
        "text": "The platform hosts over 45 models across seven categories, including chat, code, vision, image generation, audio, embeddings, and object detection. All endpoints are OpenAI SDK compatible."
    },
    {
        "id": "doc-003",
        "text": "Flagship models include Llama 3.3 70B for general-purpose tasks, DeepSeek R1 671B MoE for deep reasoning, and Kimi K2.6 for advanced agentic coding with vision support."
    },
    {
        "id": "doc-004",
        "text": "The Free tier offers 60 requests per day across more than 16 models. Pro is $80 per month for 1,000 requests per day, and Premium is $350 per month for 5,000 requests per day with priority queue access."
    }
]

3. Build a minimal retriever

Before calling the model, I need to find relevant chunks. I use a simple keyword overlap scorer. It requires zero dependencies, runs instantly, and drops cleanly into a vector database later if needed.

def retrieve_context(query, documents, top_k=2):
    query_terms = set(query.lower().split())
    scored = []
    for doc in documents:
        doc_terms = set(doc["text"].lower().split())
        overlap = len(query_terms & doc_terms)
        scored.append((overlap, doc))
    scored.sort(key=lambda x: x[0], reverse=True)
    return [doc for score, doc in scored[:top_k] if score > 0]

4. Lock in the system prompt

The system prompt is the most important part of a Q&A agent. I force the model to stay grounded in the provided context, cite sources by ID, and refuse to guess.

SYSTEM_PROMPT = """You are a precise question-answering agent.
Answer the user's question using ONLY the provided context.
If the context does not contain the answer, say "I don't have enough information to answer that."
Cite your sources by including the document ID in square brackets, like [doc-001].
Keep your answer concise and factual."""

5. Wire the answer generator

This function ties retrieval to generation. I format the top chunks into a user message and call Llama 3.3 70B through Oxlo.ai. Because Oxlo.ai uses flat per-request pricing, I can pass long context windows stuffed with documents without inflating cost.

def ask_question(query):
    context_docs = retrieve_context(query, DOCUMENTS, top_k=2)
    if not context_docs:
        return "No relevant documents found."
    
    context_block = "\n\n".join(
        f"[{doc['id']}] {doc['text']}" for doc in context_docs
    )
    
    user_message = f"Context:\n{context_block}\n\nQuestion: {query}"
    
    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
        temperature=0.1,
    )
    return response.choices[0].message.content

Run it

I test the agent with two questions. One is directly covered by the knowledge base, and the other is out of scope to verify the refusal behavior.

if __name__ == "__main__":
    print(ask_question("Which model should I use for general-purpose tasks?"))
    print("---")
    print(ask_question("What is the weather in Tokyo?"))

Example output:

For general-purpose tasks, Oxlo.ai offers Llama 3.3 70B [doc-003].
---
I don't have enough information to answer that.

Next steps

Replace the keyword retriever with embedding search using Oxlo.ai's BGE-Large or E5-Large endpoints, or switch to DeepSeek R1 671B MoE if your questions require multi-step reasoning over the context. You can also wrap the ask_question function in a FastAPI POST route to serve the agent to a frontend.