Building Question Answering Systems with LLM

Question answering systems built on large language models have become the standard interface for enterprise knowledge bases. Rather than fine-tuning models each time a document changes, modern pipelines use retrieval-augmented generation to ground responses in private data without updating model weights. The result is a flexible architecture where the LLM serves as a reasoning layer over a searchable index.

Architecture Overview

A production QA system typically splits work between two components. A retriever finds relevant passages from a document corpus, and a generator synthesizes a natural language answer conditioned on those passages. This pattern, commonly called retrieval-augmented generation, keeps the LLM from hallucinating facts while allowing it to rephrase, summarize, and compare information.

The retriever usually relies on dense embeddings. You chunk documents into passages, encode them with an embedding model, and store the vectors in a search index such as pgvector, Chroma, or FAISS. At query time, you embed the user question and retrieve the top-k nearest neighbors.

The generator then receives a system prompt, the retrieved passages, and the user question. Because the prompt includes external context, the input length can grow quickly. This is where your choice of inference provider directly affects both cost and latency.

Retrieval Implementation

Start with a simple in-memory pipeline. The following example chunks text and embeds it using an embedding endpoint.

import os
from openai import OpenAI

# Oxlo.ai supports OpenAI SDK drop-in usage
client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def get_embeddings(texts, model="bge-large"):
    response = client.embeddings.create(model=model, input=texts)
    return [item.embedding for item in response.data]

# Chunk documents
documents = [
    "Oxlo.ai offers request-based pricing for LLM inference.",
    "Unlike token-based providers, Oxlo.ai charges a flat rate per API call.",
    "This makes long-context workloads significantly cheaper."
]

chunks = documents  # In production, use a proper chunking strategy
chunk_embeddings = get_embeddings(chunks)

Store these in a vector database. For prototyping, a cosine similarity search over NumPy arrays is sufficient.

Generation with Oxlo.ai

Once you have the relevant chunks, construct a prompt that tells the model to answer using only the provided context. Oxlo.ai exposes chat/completions endpoints that are fully OpenAI SDK compatible, so you can point your existing client at https://api.oxlo.ai/v1 without changing application code.

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def retrieve(query, chunks, chunk_embeddings, top_k=3):
    q_emb = get_embeddings([query])[0]
    scores = [cosine_similarity(q_emb, c_emb) for c_emb in chunk_embeddings]
    top_indices = np.argsort(scores)[-top_k:][::-1]
    return [chunks[i] for i in top_indices]

def answer_question(query, chunks, chunk_embeddings, model="llama-3.3-70b"):
    context = "\n\n".join(retrieve(query, chunks, chunk_embeddings))
    messages = [
        {
            "role": "system",
            "content": "Answer the question using only the provided context. If the answer is not present, say you do not know."
        },
        {
            "role": "user",
            "content": f"Context:\n{context}\n\nQuestion: {query}"
        }
    ]
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0.1
    )
    return response.choices[0].message.content

query = "How does Oxlo.ai price API requests?"
print(answer_question(query, chunks, chunk_embeddings))

This pattern works with any Oxlo.ai chat model, including Llama 3.3 70B for general purpose reasoning, DeepSeek R1 671B MoE for complex multi-step questions, or Kimi K2.6 for agentic coding workflows that may need to query structured tools before answering.

Managing Cost and Context Windows

QA systems are inherently long-context workloads. Each request carries a system prompt, several retrieved paragraphs, conversation history, and the user query. On token-based providers, costs scale linearly with input length, which makes high-recall retrieval strategies expensive.

Oxlo.ai uses request-based pricing: one flat cost per API request regardless of prompt length. For QA pipelines that retrieve five to ten chunks per question, this model can be 10-100x cheaper than token-based alternatives. You can see current plans at https://oxlo.ai/pricing.

Context size is equally important. If your corpus contains lengthy technical manuals, you want models that can ingest large passages without truncation. Oxlo.ai offers DeepSeek V4 Flash with a 1 million token context window, and Kimi K2.6 with 131K context and advanced reasoning capabilities. Both support vision inputs if your documents include diagrams or screenshots. There are no cold starts on popular models, so latency remains consistent even under variable load.

Advanced Techniques

Basic vector search is only the starting point. Production systems often add:

Re-ranking. A cross-encoder or a larger embedding model re-scores the top 50 candidates from the initial retrieval step to improve precision.

Hybrid search. Combine dense embeddings with sparse retrieval such as BM25 to catch exact keyword matches that semantic search misses.

Query rewriting. Use a lightweight model to expand abbreviations or rephrase vague questions before embedding them.

Structured output. Use JSON mode to force the model to return answers with citations, enabling downstream fact-checking. Oxlo.ai supports JSON mode and function calling, so you can also attach tools that query SQL databases or internal APIs before the final answer is generated.

Multi-turn memory. For conversational QA, maintain a sliding window of prior turns in the message array. Because Oxlo.ai charges per request, adding conversation history does not inflate the per-call cost the way it would on token-based platforms.

Evaluation

A working pipeline is not necessarily an accurate one. Measure at least these three dimensions:

Faithfulness: does the answer contradict the retrieved context?

Answer relevance: does the response actually address the user question?

Context precision: did the retriever return chunks that contain the answer?

You can compute these with small, dedicated classifier models, or use an LLM-as-a-judge with a stricter model such as DeepSeek R1 671B MoE on Oxlo.ai to score outputs against a labeled evaluation set.

Conclusion

Building a question answering system is now primarily an integration task: chunking, embedding, retrieving, and generating. The differentiation lies in retrieval quality and inference economics. Oxlo.ai fits this stack naturally. Its OpenAI-compatible API means you can swap it into an existing RAG pipeline in minutes, while request-based pricing removes the cost penalty for sending long contexts to powerful models. If you are prototyping a knowledge base or scaling an agentic QA service, start with the free tier at Oxlo.ai and evaluate whether flat per-request billing aligns with your workload.