Question answering with large language models is now the default pattern for extracting structured information from unstructured data. Whether you are building an internal knowledge base, a customer support bot, or a research assistant, the core challenge remains the same: grounding the model in accurate, up-to-date context without sacrificing latency or cost. This tutorial walks through a production-ready retrieval-augmented generation pipeline using the OpenAI SDK and Oxlo.ai, a developer-first inference platform with flat per-request pricing.
Architecture Overview
Most production QA systems use Retrieval-Augmented Generation rather than fine-tuning. RAG works by retrieving relevant documents and injecting them into the prompt. This keeps answers current and reduces hallucination. Fine-tuning is useful for tuning tone or task format, but it is not a substitute for grounding.
Oxlo.ai supports both paths. You can generate embeddings with BGE-Large or E5-Large, then route the retrieved chunks to any chat model. Because Oxlo.ai charges per request rather than per token, expanding context windows to include more retrieved passages does not increase cost.
Setup and Authentication
Oxlo.ai exposes a fully OpenAI-compatible API. Change the base URL and API key, and existing code works without modification.
import openai
client = openai.OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)Install the SDK with pip install openai. No custom client is required.
Building a Basic QA Pipeline
Start with direct context injection. Assume you have already retrieved a relevant passage.
def answer_question(context: str, question: str, model: str = "llama-3.3-70b") -> str:
prompt = (
"You are a precise technical assistant. Answer the question using only the provided context. "
"If the answer is not in the context, say 'I don't know'.\n\n"
f"Context:\n{context}\n\nQuestion: {question}\nAnswer:"
)
response = client.chat.completions.create(
model=model,
messages=[{"role": "user", "content": prompt}],
temperature=0.1,
max_tokens=512
)
return response.choices[0].message.content
context = """
Oxlo.ai provides request-based pricing for LLM inference.
Unlike token-based providers, the cost per API call is flat regardless of prompt length.
This makes long-context workloads significantly cheaper.
"""
print(answer_question(context, "How does Oxlo.ai price its API?"))This pattern works with any Oxlo.ai chat model, including Qwen 3 32B for multilingual queries or DeepSeek R1 671B MoE for reasoning-heavy questions.
Adding Retrieval with Embeddings
A real system retrieves context automatically. Use Oxlo.ai's embeddings endpoint to encode your knowledge base, then search with cosine similarity.
import numpy as np
def get_embedding(text: str) -> list[float]:
resp = client.embeddings.create(
model="bge-large", # or e5-large
input=text
)
return resp.data[0].embedding
# Example knowledge base
documents = [
"Oxlo.ai offers flat per-request pricing for open-source LLMs.",
"Oxlo.ai supports 45+ models across 7 categories including vision and audio.",
"The API is fully compatible with the OpenAI SDK.",
]
doc_embeddings = [get_embedding(doc) for doc in documents]
def retrieve(question: str, top_k: int = 1) -> str:
q_emb = get_embedding(question)
similarities = [
np.dot(q_emb, d_emb) / (np.linalg.norm(q_emb) * np.linalg.norm(d_emb))
for d_emb in doc_embeddings
]
best_idx = int(np.argmax(similarities))
return documents[best_idx]
question = "What models does Oxlo.ai support?"
context = retrieve(question)
print(answer_question(context, question))For production, replace the in-memory list with a vector database such as pgvector, Pinecone, or Weaviate. The embedding generation step remains identical because the endpoint follows the standard OpenAI shape.
Long Context and Reasoning
As your knowledge base grows, you may need to inject dozens of passages. On token-based providers, long prompts inflate cost linearly. Oxlo.ai uses request-based pricing, so sending a full page of retrieved documents costs the same as a one-line greeting. This makes aggressive retrieval strategies practical.
If you need to reason over very large contexts, select a model built for it. DeepSeek V4 Flash supports a 1 million token context window and efficient MoE inference. Kimi K2.6 handles 131K contexts with advanced reasoning and vision. For pure chain-of-thought depth, Kimi K2 Thinking or DeepSeek R1 671B MoE are strong candidates.
# Example: multi-passage retrieval with a long-context model
long_context = "\n\n".join([retrieve(q, top_k=3) for q in sub_questions])
response = client.chat.completions.create(
model="deepseek-v4-flash",
messages=[{"role": "user", "content": f"Answer based on the following passages:\n\n{long_context}\n\nQuestion: {question}"}],
temperature=0.2
)Evaluating Your QA System
Accuracy depends on both retrieval precision and generation faithfulness. Measure retrieval with hit rate at k. Measure generation with exact match or LLM-as-a-judge. Oxlo.ai's flat pricing lets you run large evaluation batches without worrying about token burn from repeated long prompts.
Use JSON mode to get structured evaluation outputs.
eval_prompt = (
"Evaluate whether the answer is supported by the context. "
"Respond with JSON containing 'supported' (bool) and 'reason' (str)."
)
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": eval_prompt}],
response_format={"type": "json_object"}
)Deployment Considerations
For live applications, enable streaming to improve perceived latency. Oxlo.ai supports standard SSE streaming through the same SDK.
stream = client.chat.completions.create(
model="qwen3-32b",
messages=[{"role": "user", "content": question}],
stream=True
)
for chunk in stream:
print(chunk.choices[0].delta.content or "", end="")If your QA agent needs to call external APIs, use function calling. Oxlo.ai supports tool use on compatible models such as Llama 3.3 70B, Qwen 3 32B, and Minimax M2.5. This lets the model decide when to search a database, run a calculator, or trigger a webhook.
Finally, consider the free tier for prototyping. Oxlo.ai offers 60 requests per day across 16+ models, which is enough to iterate on retrieval logic before moving to a paid plan. See https://oxlo.ai/pricing for plan details.
Conclusion
Building a reliable QA system requires clean retrieval, faithful generation, and cost-efficient inference. By combining embeddings, vector search, and a compatible chat completions API, you can ship a production pipeline in an afternoon. Oxlo.ai's request-based pricing removes the penalty for long prompts, and its OpenAI SDK compatibility means you can switch your endpoint without rewriting client code. Point your base URL to https://api.oxlo.ai/v1 and start building.
Top comments (0)