Building Question Answering Systems with LLMs

A production question answering system is more than a prompt wrapper around a large language model. It requires a retrieval pipeline that surfaces relevant context, a generation stage that synthesizes answers without hallucination, and an inference backend that remains cost-effective when context windows grow. Teams building these systems today face a familiar tradeoff. They can constrain context length to save on token-based billing, or they can pay linearly for every extra sentence they feed into the model. A request-based pricing model changes that calculus entirely.

Architecture of a Modern QA System

Most production implementations follow retrieval-augmented generation, or RAG. The pipeline has three phases: ingestion, retrieval, and synthesis.

During ingestion, source documents are split into chunks, converted to dense vectors using an embedding model, and indexed in a vector database. At query time, the user question is embedded with the same model, and the top-k nearest neighbor chunks are retrieved. These chunks, along with the original question and any system instructions, form the prompt for the LLM.

The choice of embedding model affects recall. Oxlo.ai offers BGE-Large and E5-Large through a standard embeddings endpoint, so you can keep the vector generation stage on the same platform as the generation stage. Both models are suitable for technical documentation and general knowledge bases.

Retrieval Strategy and Chunking

Chunk size and overlap are hyperparameters. Smaller chunks improve precision but may lose surrounding context. A common starting point is 512 tokens with 128 tokens of overlap, though this depends on document structure.

Hybrid search often outperforms pure vector search. Consider combining dense retrieval with keyword matching through BM25 or sparse embeddings. The retrieved chunks should include metadata, such as source URLs or page numbers, so the generation stage can cite its sources.

Generation and Model Selection

The LLM must read the retrieved context and produce a concise, accurate answer. Model selection depends on the domain.

For general-purpose QA over broad knowledge bases, Llama 3.3 70B provides strong instruction following and factual grounding. If the system must reason over multiple documents or perform multi-step synthesis, DeepSeek R1 671B MoE or Kimi K2.6 offer advanced chain-of-thought capabilities. For developer documentation or API references, Qwen 3 Coder 30B and DeepSeek Coder are purpose-built options. Oxlo.ai hosts all of these models behind a single OpenAI-compatible endpoint, so switching between them is a one-line parameter change.

The Cost of Context in QA Workloads

Question answering systems are natural long-context workloads. A single user query might pull ten document chunks, each several hundred tokens, plus a detailed system prompt. Under token-based pricing, input costs scale linearly with every chunk you include. Many teams respond by trimming context or reducing chunk count, which directly hurts answer quality.

Oxlo.ai uses request-based pricing: one flat cost per API call regardless of prompt length. This means you can include more retrieved chunks, add few-shot examples, or pass lengthy system instructions without watching inference costs climb. For agentic QA systems that iterate through tool calls and accumulate conversation history, the savings compound. Models like DeepSeek V4 Flash, with its 1 million token context window, and Kimi K2.6, with 131K context and vision support, become practical choices rather than expensive experiments.

Code Example: Querying Oxlo.ai with Retrieved Context

Because Oxlo.ai is fully compatible with the OpenAI SDK, you can integrate it into an existing RAG pipeline by changing the base URL and model name.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

def answer_question(question: str, context_chunks: list[str]) -> str:
    context = "\n\n".join(context_chunks)

    response = client.chat.completions.create(
        model="llama-3.3-70b",  # or deepseek-r1-671b, qwen-3-32b, etc.
        messages=[
            {
                "role": "system",
                "content": (
                    "You are a precise technical assistant. "
                    "Answer the question using only the provided context. "
                    "If the answer is not in the context, say you do not know."
                )
            },
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {question}"
            }
        ],
        temperature=0.1,
        stream=False
    )

    return response.choices[0].message.content

# Example usage
chunks = [
    "Oxlo.ai provides request-based pricing for LLM inference.",
    "The platform supports 45+ models across seven categories.",
    "There are no cold starts on popular models."
]

print(answer_question("How does Oxlo.ai price its API?", chunks))

The same client supports streaming responses, JSON mode, and function calling. If your QA system needs to call external APIs to resolve a query, you can enable tool use without switching libraries.

Evaluation and Iteration

Shipping a QA system requires more than a working prototype. You need an evaluation framework that measures both retrieval accuracy and generation quality.

Start with a labeled dataset of question-context-answer triples. Measure retrieval with hit rate and mean reciprocal rank. Measure generation with semantic similarity between predicted and reference answers, or use an LLM-as-judge approach with a stronger model like GLM 5 or Kimi K2 Thinking to score faithfulness and relevance.

Trace individual requests. Oxlo.ai offers streaming and standard chat completions endpoints, so you can log latency and token usage through your existing observability stack. Because pricing is per request, cost forecasting is straightforward: multiply your expected daily query volume by your plan's request limit or rate, rather than estimating average tokens per query.

Conclusion

Building a reliable question answering system means optimizing across retrieval, generation, and cost. Long context is not a luxury in RAG, it is a requirement for accuracy. Token-based billing forces teams to compromise on context length. Oxlo.ai's request-based pricing removes that constraint, letting you send full retrieved documents and conversation history at a predictable cost. With 45+ models, OpenAI SDK compatibility, and no cold starts, it is a practical backend for QA systems that need to scale without surprise bills. Review the pricing and model catalog at https://oxlo.ai/pricing to see how per-request billing fits your workload.