Building a Question Answering System using LLM

#aiinfrastructure #oxlo #ai

Question answering is one of the most common production workloads for large language models. Whether you are building an internal knowledge base, a customer support bot, or a legal research assistant, the core challenge is identical: grounding a generative model in accurate, up-to-date context and returning a precise, verifiable answer. Modern open-source models and inference platforms have made this architecture accessible, but the choice of backend determines whether your QA pipeline remains cost-effective at scale.

Architecture Patterns for LLM QA

Most production QA systems rely on Retrieval-Augmented Generation, or RAG. Instead of expecting the model to memorize facts, you retrieve relevant documents at query time and inject them into the prompt. A typical pipeline has four stages: ingestion, embedding, retrieval, and generation. For ingestion, documents are chunked and converted into vector embeddings. During retrieval, a user query is embedded and matched against the vector store. Finally, a language model generates an answer conditioned on the retrieved chunks.

Some workloads skip retrieval and use a model's parametric knowledge alone, but this risks hallucination and stale information. For grounded QA, RAG is the standard pattern. The generation stage is where your inference provider matters most, because latency, context-window size, and pricing all affect user experience.

Implementation: A Minimal RAG Pipeline

Below is a concise Python example that embeds a query, retrieves context, and generates an answer. It uses the OpenAI SDK with Oxlo.ai as the provider, which means you can switch models or endpoints without rewriting client code.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

def get_embedding(text: str) -> list[float]:
    response = client.embeddings.create(
        model="bge-large",
        input=text
    )
    return response.data[0].embedding

def answer_question(context: str, question: str, model: str = "llama-3.3-70b") -> str:
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "system", "content": "You are a precise QA assistant. Answer using only the provided context. If the answer is not in the context, say you do not know."},
            {"role": "user", "content": f"Context:\n{context}\n\nQuestion: {question}"}
        ],
        temperature=0.1,
        stream=False
    )
    return response.choices[0].message.content

# Example usage
context = "Oxlo.ai offers flat per-request pricing for open-source LLMs. Unlike token-based providers, cost does not scale with prompt length."
question = "How does Oxlo.ai price API requests?"
print(answer_question(context, question))

This pattern scales to any vector store. The critical detail is that the chat completion call is fully OpenAI API compatible, so you can drop Oxlo.ai into existing RAG frameworks such as LangChain or LlamaIndex without adapter layers.

Selecting Models and Infrastructure

The model you choose should match the complexity and language of your questions. For general-purpose QA, Llama 3.3 70B provides a strong balance of reasoning speed and accuracy. If your pipeline involves deep reasoning over technical documentation or code, DeepSeek R1 671B MoE or Kimi K2.6 are better fits. Qwen 3 32B excels at multilingual agent workflows, while DeepSeek V4 Flash offers a one-million-token context window for scenarios where you must feed entire manuals into a single request.

Oxlo.ai hosts these models and more than 45 others across seven categories, including dedicated embedding models such as BGE-Large and E5-Large, vision models like Gemma 3 27B for document understanding, and audio models for speech-based Q&A. Because the platform supports function calling, JSON mode, and streaming, you can build agentic QA systems that cite sources, validate facts against external APIs, or return structured answers.

Cost Engineering for Long-Context QA

Question answering is expensive when you are billed by the token. Every retrieved document chunk added to the prompt increases input cost, and long-context models can ingest hundreds of thousands of tokens per request. Token-based providers, including Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, scale cost linearly with prompt length. For a high-volume QA system, this pricing model penalizes accuracy, because retrieving more context to reduce hallucination directly raises your bill.

Oxlo.ai uses request-based pricing: one flat cost per API call regardless of prompt length. For long-context and agentic QA workloads, this can be significantly cheaper than token-based alternatives. You can retrieve generously, include system prompts, and maintain multi-turn history without watching metered tokens accumulate. See the Oxlo.ai pricing page for current plan details.

Moving to Production

Once your pipeline is functional, focus on latency and reliability. Streaming responses improve perceived speed for end users. JSON mode lets you enforce structured output schemas for answers and citations. If your application requires follow-up clarification, multi-turn conversation support keeps context coherent without manual prompt stitching.

Cold starts can break user trust in interactive QA. Oxlo.ai keeps popular models warm, so you do not see sporadic latency spikes on first requests. When evaluating providers, run load tests that include long prompts and measure time-to-first-token across different times of day. Predictable performance matters as much as benchmark scores for production QA.

DEV Community