Question answering systems are one of the most common production workloads for large language models. Whether you are building an internal knowledge base, a customer support agent, or a research assistant, the core challenge remains the same: retrieving accurate information and synthesizing it into a concise, correct response. This guide walks through the architectural decisions, model selection, and implementation patterns you need to build a reliable QA system, using Oxlo.ai as the inference backend.
Architecture Overview
Most production QA systems fall into two categories. Closed-book systems rely entirely on the model's parametric knowledge. Open-book systems, typically built with retrieval-augmented generation (RAG), inject external text into the prompt at inference time. The open-book approach reduces hallucinations and keeps answers current, but it increases context length and requires an embedding model for retrieval. Both patterns benefit from an inference layer that handles long inputs efficiently.
Choosing a Model
Oxlo.ai hosts over 45 open-source and proprietary models across seven categories, so you can match the model to your QA workload rather than forcing every question through a single endpoint.
- General-purpose QA: Llama 3.3 70B provides a strong balance of instruction following and latency for standard question answering.
- Reasoning and coding: DeepSeek R1 671B MoE and Kimi K2.6 excel at multi-step reasoning, math, and complex coding questions.
- Multilingual workloads: Qwen 3 32B handles non-English documents and cross-lingual retrieval well.
- Embeddings: BGE-Large and E5-Large are available for vectorizing documents and queries in a RAG pipeline.
All models are served with no cold starts, and the platform is fully OpenAI SDK compatible.
Setting Up the API Client
Because Oxlo.ai exposes an OpenAI-compatible API at https://api.oxlo.ai/v1, you can use the official Python SDK with a two-line change.
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)This client will handle chat completions, embeddings, and streaming responses without additional adapters.
Closed-Book Question Answering
For questions that fall within the model's training data, a single chat completion is sufficient. The following example uses JSON mode to enforce a structured answer with a confidence score.
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a precise QA assistant. Answer in JSON with keys: answer, confidence."},
{"role": "user", "content": "What is the capital of Mongolia?"}
],
response_format={"type": "json_object"}
)
print(response.choices[0].message.content)JSON mode is supported across Oxlo.ai's chat models, which makes it easy to parse outputs in downstream pipelines.
Open-Book QA and Retrieval-Augmented Generation
When answers must come from private documents, use an embedding model to retrieve relevant passages, then inject them into the prompt. Oxlo.ai offers BGE-Large and E5-Large for this step.
# 1. Embed the user's question
question = "What are the termination clauses in the 2024 vendor agreement?"
q_embedding = client.embeddings.create(
model="bge-large",
input=question
).data[0].embedding
# 2. Retrieve top-k chunks from your vector store (implementation omitted)
context_chunks = vector_store.search(q_embedding, top_k=3)
context = "\n\n".join(context_chunks)
# 3. Generate an answer grounded in the retrieved context
response = client.chat.completions.create(
model="deepseek-r1-671b",
messages=[
{"role": "system", "content": "Answer using only the provided context. Cite the source chunk number."},
{"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
]
)
print(response.choices[0].message.content)For agentic workflows, you can combine this pattern with function calling so the model decides whether to query a vector store, call an API, or answer directly.
Leveraging Long Context
One of the biggest cost drivers in token-based inference is input length. Every retrieved chunk, system prompt, and conversation history token adds to the bill. Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. This makes long-context workloads, such as dumping an entire legal brief or a full repository of documentation into the prompt, significantly more predictable.
Models like DeepSeek V4 Flash support a 1 million token context window, and Kimi K2.6 offers a 131K context window with advanced reasoning and vision. On Oxlo.ai, you can use these extended contexts without watching input tokens scale your costs linearly. For current plan details, see https://oxlo.ai/pricing.
Structured Output and Tooling
Reliable QA systems rarely return raw prose. Oxlo.ai supports streaming, function calling, and JSON mode, which lets you build pipelines that emit structured answers, trigger follow-up actions, or stream partial results to the user.
tools = [
{
"type": "function",
"function": {
"name": "search_knowledge_base",
"description": "Search internal docs for a query",
"parameters": {
"type": "object",
"properties": {
"query": {"type": "string"}
},
"required": ["query"]
}
}
}
]
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[{"role": "user", "content": "How do I reset the production database?"}],
tools=tools,
tool_choice="auto"
)If the model invokes search_knowledge_base, your application can execute the retrieval and return the results in a follow-up turn.
Putting It Together
A production QA system on Oxlo.ai typically looks like this: embed queries with BGE-Large or E5-Large, retrieve relevant context, route complex reasoning to DeepSeek R1 or Kimi K2.6, and stream structured JSON answers back to the client. Because the platform is a drop-in replacement for the OpenAI SDK, you can prototype on the Free tier, which includes 60 requests per day and access to more than 16 free models, then move to Pro or Premium as traffic grows.
Request-based pricing removes the penalty for long prompts, so you can focus on answer quality rather than token economy. To explore plans and model availability, visit https://oxlo.ai/pricing.
Top comments (0)