Using LLM for Question Answering and Information Retrieval

We will build a minimal retrieval-augmented QA agent that answers questions over a small internal knowledge base. It retrieves relevant text chunks with a basic inverted index and then generates an answer with an LLM. This pattern is useful for support bots, internal wikis, or any domain where you cannot trust the model to hallucinate facts.

What you'll need

• Python 3.10 or newer
• An Oxlo.ai API key from https://portal.oxlo.ai
• The OpenAI SDK: pip install openai

Step 1: Ingest and index the documents

We will hardcode four tiny internal docs about Oxlo.ai, clean them, and build a word-level inverted index so we can retrieve candidates quickly. This keeps the tutorial self-contained and avoids external dependencies.

import re
from collections import defaultdict

DOCS = [
    "Oxlo.ai Authentication: All API requests require a key passed in the Authorization header as Bearer YOUR_OXLO_API_KEY.",
    "Oxlo.ai Rate Limits: Free plans allow 60 requests per day. Pro plans allow 1,000 requests per day. Premium plans allow 5,000 requests per day.",
    "Oxlo.ai SDK Setup: Install the OpenAI SDK with pip install openai. Set base_url to https://api.oxlo.ai/v1 and use your Oxlo.ai API key.",
    "Oxlo.ai Models: Available models include llama-3.3-70b for general tasks, qwen-3-32b for multilingual reasoning, deepseek-v3.2 for coding and reasoning, and kimi-k2.6 for agentic coding."
]

def tokenize(text):
    return set(re.findall(r"\b[a-z0-9.]+\b", text.lower()))

index = defaultdict(set)
for i, doc in enumerate(DOCS):
    for tok in tokenize(doc):
        index[tok].add(i)

print(f"Indexed {len(DOCS)} documents with {len(index)} unique terms.")

Step 2: Build the retriever

The retriever scores documents by counting overlapping terms with the query. We return the top two chunks to keep the final prompt concise.

from collections import Counter

def retrieve(query, k=2):
    q_tokens = tokenize(query)
    scores = Counter()
    for tok in q_tokens:
        for doc_id in index.get(tok, []):
            scores[doc_id] += 1
    # Break ties by preferring shorter docs
    top_ids = sorted(scores.keys(), key=lambda i: (-scores[i], len(DOCS[i])))[:k]
    return [DOCS[i] for i in top_ids]

# Smoke test
print(retrieve("How do I authenticate?"))

Step 3: Write the system prompt

The system prompt restricts the model to the provided context and requires it to cite its source. We leave a {context} placeholder so we can inject retrieved chunks at runtime.

SYSTEM_PROMPT_TEMPLATE = """You are a precise technical support agent.
Answer the user's question using ONLY the retrieved context below.
If the context does not contain the answer, say "I don't have that information."
Cite the source by quoting the document title briefly.

Retrieved context:
{context}
"""

Step 4: Wire the answer generator

Now we combine retrieval and generation. Because Oxlo.ai uses request-based pricing, adding retrieved text to the prompt does not increase the cost of the call, which makes RAG workloads predictable. You can explore pricing at https://oxlo.ai/pricing.

from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def answer_question(question):
    chunks = retrieve(question, k=2)
    context = "\n\n".join(f"- {c}" for c in chunks)
    SYSTEM_PROMPT = SYSTEM_PROMPT_TEMPLATE.format(context=context)

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": question},
        ],
        temperature=0.1,
    )
    return response.choices[0].message.content.strip()

print(answer_question("What are the rate limits on the free plan?"))

Run it

Here is a short test loop and the output you should see after replacing YOUR_OXLO_API_KEY with a real key.

QUESTIONS = [
    "How do I authenticate?",
    "Which model should I use for coding?",
    "What is the enterprise SLA?",
]

for q in QUESTIONS:
    print(f"Q: {q}")
    print(f"A: {answer_question(q)}")
    print()

Q: How do I authenticate?
A: You must pass your API key in the Authorization header as a Bearer token. Source: Oxlo.ai Authentication.

Q: Which model should I use for coding?
A: For coding tasks, you can use kimi-k2.6, which is designed for agentic coding, or deepseek-v3.2 for coding and reasoning. Source: Oxlo.ai Models.

Q: What is the enterprise SLA?
A: I don't have that information.

Next steps

Swap the inverted index for dense embeddings with Oxlo.ai's BGE-Large or E5-Large models to handle paraphrased questions. You could also add a re-ranking step with deepseek-v3.2 or kimi-k2.6 to drop irrelevant chunks before generation. Because Oxlo.ai bills per request rather than per token, longer prompts for reranking or large context windows do not inflate your costs.

Comments

[Luis]

I appreciate how you've structured the retriever to score documents based on overlapping terms with the query, and then return the top two chunks to keep the final prompt concise. The use of a basic inverted index and the tokenize function to preprocess the documents is a good approach for a minimal retrieval-augmented QA agent. One potential improvement could be to explore more advanced techniques for document ranking, such as TF-IDF or BM25, to further improve the accuracy of the retriever. Have you considered experimenting with these alternatives, and if so, what were your findings?