A production question answering system is rarely just a prompt and a model. You need retrieval to ground answers in private data, a generator that can reason across long documents, and an inference backend that does not punish you for large inputs. Most tutorials default to token-based billing, which makes long-context QA expensive. Oxlo.ai uses flat per-request pricing, so the cost of stuffing a retrieved knowledge base into the prompt is the same as a one-sentence query. This changes how you architect the retrieval step and which models you can afford to run.
Architecture Overview
Most LLM QA systems follow one of two patterns. Retrieval-Augmented Generation, or RAG, embeds documents into a vector database and retrieves the top-k relevant chunks before generation. Long-context direct inference skips retrieval and places entire documents, or even corpora, directly into the model's context window. A hybrid approach retrieves first, then re-ranks and injects a large subset of results into a long-context model for final synthesis.
Your choice depends on latency, corpus size, and update frequency. RAG is essential when data changes often. Long-context inference is simpler when you have static manuals or legal contracts and a model with a large enough window. Oxlo.ai carries both DeepSeek V4 Flash, which offers a 1M context window, and Kimi K2.6 with a 131K context window, so you are not forced into a complex retrieval pipeline just to avoid token costs.
Model Selection
For factual extraction over retrieved text, a capable general-purpose model is usually sufficient. On Oxlo.ai, Llama 3.3 70B is a solid default for low-latency answers. If the question requires multi-step reasoning, such as comparing clauses across documents or performing arithmetic on tabular data, switch to a reasoning specialist. DeepSeek R1 671B MoE, Kimi K2.6, or Qwen 3 32B all handle chain-of-thought reasoning well. Because Oxlo.ai bills per request rather than per token, you can send lengthy reasoning prompts or receive long chain-of-thought outputs without watching metered costs scale.
Retrieval and Embeddings
If you use RAG, you need an embedding model to convert text into vectors. Oxlo.ai provides BGE-Large and E5-Large through the standard OpenAI-compatible embeddings endpoint. You can store vectors in any database, such as pgvector, Chroma, or Pinecone, then run cosine similarity at query time.
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
def get_embedding(text):
response = client.embeddings.create(
model="bge-large",
input=[text]
)
return response.data[0].embedding
Index your documents in batches, keep the chunk size between 256 and 512 tokens with overlap, and store metadata so you can cite sources in the final answer.
Generation with Oxlo.ai
The chat completions endpoint is a drop-in replacement for the OpenAI SDK. After retrieving context chunks, inject them into the system or user message and ask the model to answer based strictly on the provided text. Streaming is supported, so you can return tokens to the user as they are generated.
def answer_question(question, contexts):
context_block = "\n---\n".join(contexts)
prompt = f"Context:\n{context_block}\n\nQuestion: {question}"
stream = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a precise QA assistant. Answer using only the provided context."},
{"role": "user", "content": prompt}
],
stream=True
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")
This pattern works with any Oxlo.ai chat model. You can switch to Qwen 3 32B for multilingual documents, or DeepSeek R1 671B MoE for reasoning-heavy questions, without changing any client code.
Long-Context QA
Token-based billing creates a disincentive to use large context windows. When every input token costs money, stuffing ten retrieved passages into a prompt feels like a budget risk. Oxlo.ai removes that friction. The platform charges one flat rate per API request, regardless of prompt length. This means a 1,000-token question and a 100,000-token document analysis cost the same.
For internal knowledge bases or legal discovery, you can often fit an entire document set into DeepSeek V4 Flash's 1M context window and ask questions directly. You eliminate retrieval latency, reduce architecture complexity, and still pay a single request fee. If your corpus is larger than one context window, use retrieval to narrow down to a relevant subset, then inject that subset into a long-context model for synthesis. Either way, your costs remain predictable.
Agentic Tool Use
Some questions cannot be answered from static text alone. If a user asks for the current server status or wants to run a calculation on private data, the model needs tools. Oxlo.ai supports function calling and tool use across compatible models, including Qwen 3 32B and Minimax M2.5.
tools = [
{
"type": "function",
"function": {
"name": "query_database",
"description": "Run a SQL query against the internal analytics database",
"parameters": {
"type": "object",
"properties": {
"sql": {"type": "string"}
},
"required": ["sql"]
}
}
}
]
response = client.chat.completions.create(
model="qwen-3-32b",
messages=[{"role": "user", "content": "What were yesterday's top errors?"}],
tools=tools
)
The model can decide to call the function, receive the result, and generate a final answer in a multi-turn conversation. This turns a simple QA bot into an agent that can reason over live data.
Putting It Together
A complete pipeline looks like this. First, chunk and embed your documents using Oxlo.ai's embeddings endpoint. Second, store vectors and metadata. Third, at query time, embed the question, retrieve the top-k chunks, and build a context block. Fourth, send the context and question to an Oxlo.ai chat model with streaming enabled. Fifth, if the question requires live data, define tools and let the model call them.
Because Oxlo.ai is fully OpenAI SDK compatible, you can prototype with your existing code by changing only the base URL and API key. There are no cold starts on popular models, so the first request of the day returns as fast as the hundredth.
Conclusion
Building a question answering system is straightforward, but cost structure shapes architecture. Token-based pricing pushes developers toward smaller prompts and complex retrieval layers. Oxlo.ai's flat per-request pricing lets you use long-context models and large prompt stuffing without budget surprises. With 45+ models, embedding endpoints, tool use, and streaming, Oxlo.ai gives you the infrastructure to build RAG, long-context, or agentic QA systems under one consistent API. For details on request limits and plans, see https://oxlo.ai/pricing.
The
Top comments (0)