DEV Community

shashank ms
shashank ms

Posted on

Leveraging LLM for Question Answering and Information Retrieval

#learnai #oxlo #ai

We are going to build a minimal retrieval-augmented generation pipeline that answers questions over a custom text corpus. This pattern is useful for internal support portals, legal discovery, or any domain where a generic model is likely to hallucinate dates and figures. Instead of trusting the LLM's parametric memory, we retrieve the relevant snippets first, then generate an answer grounded in that context.

What you'll need

  • Python 3.10 or newer
  • pip install openai numpy
  • An Oxlo.ai API key from https://portal.oxlo.ai
  • A few paragraphs of source text. I include a sample knowledge base below so you can run the script immediately.

Step 1: Chunk the source text

I split the raw text into overlapping chunks of roughly 200 characters. Small chunks improve retrieval precision because the embedding model can focus on a single fact instead of an entire page.

RAW_TEXT = """
Oxlo.ai is a developer-first AI inference platform with request-based pricing.
One flat cost per API request regardless of prompt length.
Unlike token-based providers, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context workloads.
The platform offers 45+ open-source and proprietary models across 7 categories, fully OpenAI SDK compatible, with no cold starts.
Flagship models include Llama 3.3 70B, Qwen 3 32B, DeepSeek R1 671B MoE, and Kimi K2.6.
"""

def chunk_text(text, size=200, overlap=50):
    chunks = []
    start = 0
    while start < len(text):
        chunks.append(text[start:start + size].strip())
        start += size - overlap
    return [c for c in chunks if c]

chunks = chunk_text(RAW_TEXT)
print(f"Created {len(chunks)} chunks")

Step 2: Embed the chunks with Oxlo.ai

I send each chunk to the Oxlo.ai embeddings endpoint using the BGE-Large model. The resulting vectors are stored in a NumPy array for fast similarity search.

import numpy as np
from openai import OpenAI

client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

def embed(texts):
    response = client.embeddings.create(
        model="bge-large",
        input=texts,
    )
    return [item.embedding for item in response.data]

chunk_embeddings = embed(chunks)
chunk_matrix = np.array(chunk_embeddings)
print(f"Embedding matrix shape: {chunk_matrix.shape}")

Step 3: Retrieve relevant chunks

I embed the user question, normalize both sets of vectors, and compute cosine similarity via a dot product. The top two chunks become the context for the generator.

def retrieve(query, top_k=2):
    q_emb = embed([query])[0]
    q_vec = np.array(q_emb)

    similarities = chunk_matrix.dot(q_vec) / (
        np.linalg.norm(chunk_matrix, axis=1) * np.linalg.norm(q_vec)
    )
    top_indices = np.argsort(similarities)[-top_k:][::-1]
    return [chunks[i] for i in top_indices]

question = "How does Oxlo.ai pricing work?"
retrieved = retrieve(question)
print("Retrieved chunks:")
for i, text in enumerate(retrieved, 1):
    print(f"{i}. {text}")

Step 4: Define the system prompt

The system prompt constrains the model to use only the retrieved context and to say "I do not know" when the answer is not present. This prevents hallucination.

SYSTEM_PROMPT = """You are a precise question-answering assistant.

Answer the user's question using ONLY the provided context.
If the context does not contain the answer, reply exactly: "I do not know."
Be concise. Do not add speculation.
"""

Step 5: Generate the grounded answer

I concatenate the retrieved chunks into a user message and call Llama 3.3 70B through Oxlo.ai. Because Oxlo.ai uses flat per-request pricing, stuffing long retrieved context into the prompt does not inflate cost the way token-based pricing would.

def answer_question(query):
    context = "\n---\n".join(retrieve(query))
    user_message = f"Context:\n{context}\n\nQuestion: {query}"

    from openai import OpenAI

    client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")

    response = client.chat.completions.create(
        model="llama-3.3-70b",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": user_message},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(answer_question("How does Oxlo.ai pricing work?"))

Run it

Save all the code above into a single file named rag.py, replace YOUR_OXLO_API_KEY, and run python rag.py. Here is the output on my machine.

$ python rag.py
Created 3 chunks
Embedding matrix shape: (3, 1024)
Retrieved chunks:
1. Oxlo.ai is a developer-first AI inference platform with request-based pricing. One flat cost per API request regardless of prompt length.
2. Unlike token-based providers, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context workloads.

Answer:
Oxlo.ai uses request-based pricing, which means you pay one flat cost per API request no matter how long the prompt is.

Next steps

Swap the generator to Kimi K2.6 if you need its 131K context window to stuff dozens of retrieved chunks into a single prompt without truncation. You can also try DeepSeek V3.2 to keep the experiment on the free tier while you iterate. If you move to production, replace the in-memory NumPy index with a proper vector database such as pgvector or Milvus, and consider adding a re-ranking step before the LLM call.

Top comments (0)