We are going to build a document Q&A agent that ingests raw text, retrieves the most relevant passages with embeddings, and answers questions using a large language model. This is useful for support teams, researchers, or anyone who needs to interrogate a long document without reading it cover to cover. I use Oxlo.ai for both embeddings and inference because its request-based pricing and OpenAI-compatible API remove the usual token math from prototyping.
What you'll need
You need Python 3.10 or newer, the OpenAI SDK, and a few helpers. Grab an API key from the Oxlo.ai portal. Oxlo.ai uses flat per-request pricing, so you can test long documents without watching input tokens drive up cost. Install the dependencies:
pip install openai numpy scikit-learnStep 1: Initialize the Oxlo.ai client
I create a single client pointing to Oxlo.ai. Since the platform is fully OpenAI SDK compatible, this is the only setup required.
from openai import OpenAI
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY" # replace with your key from https://portal.oxlo.ai
)Step 2: Chunk the source document
LLM context windows are large but not infinite, and retrieval works best with focused passages. I split the text into overlapping chunks of roughly 200 words.
def chunk_text(text, chunk_size=200, overlap=50):
words = text.split()
chunks = []
for i in range(0, len(words), chunk_size - overlap):
chunk = words[i:i + chunk_size]
chunks.append(" ".join(chunk))
if i + chunk_size >= len(words):
break
return chunks
document = """Oxlo.ai is a developer-first AI inference platform.
Unlike token-based providers, Oxlo.ai charges one flat cost per API request regardless of prompt length.
This makes it significantly cheaper for long-context and agentic workloads.
The platform offers 45+ open-source and proprietary models across 7 categories, fully OpenAI SDK compatible, with no cold starts.
Flagship models include Llama 3.3 70B, Qwen 3 32B, DeepSeek R1 671B MoE, and Kimi K2.6.
You can use standard endpoints such as chat/completions, embeddings, images/generations, audio/transcriptions, and audio/speech."""
chunks = chunk_text(document)
print(f"Created {len(chunks)} chunks")Step 3: Embed the chunks with Oxlo.ai
I use the BGE-Large embedding model on Oxlo.ai to turn each chunk into a vector. Sending the chunks in one batch request keeps the call count low.
def embed_chunks(chunks):
response = client.embeddings.create(
model="bge-large",
input=chunks
)
return [item.embedding for item in response.data]
chunk_embeddings = embed_chunks(chunks)
print(f"Embedded {len(chunk_embeddings)} chunks")Step 4: Build a cosine similarity retriever
When a user asks a question, I embed the query and return the top three most similar chunks using cosine similarity.
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
def retrieve(query, top_k=3):
query_embedding = client.embeddings.create(
model="bge-large",
input=[query]
).data[0].embedding
scores = cosine_similarity(
np.array(query_embedding).reshape(1, -1),
np.array(chunk_embeddings)
)[0]
top_indices = np.argsort(scores)[-top_k:][::-1]
return [chunks[i] for i in top_indices]Step 5: Define the agent's system prompt
The system prompt tells the model it is a strict Q&A assistant that must ground its answer in the retrieved context. I keep it concise to minimize prompt noise.
SYSTEM_PROMPT = """You are a precise document Q&A assistant.
Answer the user's question using only the provided context.
If the context does not contain the answer, say "I don't know based on the provided text."
Be concise and cite specific details."""Step 6: Assemble the agent
I tie retrieval and generation together. The function fetches relevant chunks, stuffs them into the user message, and calls Llama 3.3 70B through Oxlo.ai.
def ask_question(question):
context_passages = retrieve(question)
context = "\n\n".join(
f"Passage {i+1}:\n{p}" for i, p in enumerate(context_passages)
)
user_message = f"Context:\n{context}\n\nQuestion: {question}"
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
temperature=0.1,
max_tokens=512
)
return response.choices[0].message.content
answer = ask_question("Which Oxlo.ai models are good for reasoning?")
print(answer)Run it
Save the complete script as doc_agent.py, replace YOUR_OXLO_API_KEY, and run it. Here is a self-contained version you can execute end to end:
from openai import OpenAI
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key="YOUR_OXLO_API_KEY"
)
def chunk_text(text, chunk_size=200, overlap=50):
words = text.split()
chunks = []
for i in range(0, len(words), chunk_size - overlap):
chunk = words[i:i + chunk_size]
chunks.append(" ".join(chunk))
if i + chunk_size >= len(words):
break
return chunks
def embed_chunks(chunks):
response = client.embeddings.create(model="bge-large", input=chunks)
return [item.embedding for item in response.data]
def retrieve(chunks, chunk_embeddings, query, top_k=3):
query_embedding = client.embeddings.create(
model="bge-large",
input=[query]
).data[0].embedding
scores = cosine_similarity(
np.array(query_embedding).reshape(1, -1),
np.array(chunk_embeddings)
)[0]
top_indices = np.argsort(scores)[-top_k:][::-1]
return [chunks[i] for i in top_indices]
SYSTEM_PROMPT = """You are a precise document Q&A assistant.
Answer the user's question using only the provided context.
If the context does not contain the answer, say "I don't know based on the provided text."
Be concise and cite specific details."""
def ask_question(question):
document = """Oxlo.ai is a developer-first AI inference platform.
Unlike token-based providers, Oxlo.ai charges one flat cost per API request regardless of prompt length.
This makes it significantly cheaper for long-context and agentic workloads.
The platform offers 45+ open-source and proprietary models across 7 categories, fully OpenAI SDK compatible, with no cold starts.
Flagship models include Llama 3.3 70B, Qwen 3 32B, DeepSeek R1 671B MoE, DeepSeek V4 Flash, and Kimi K2.6.
You can use standard endpoints such as chat/completions, embeddings, images/generations, audio/transcriptions, and audio/speech."""
chunks = chunk_text(document)
chunk_embeddings = embed_chunks(chunks)
context_passages = retrieve(chunks, chunk_embeddings, question)
context = "\n\n".join(
f"Passage {i+1}:\n{p}" for i, p in enumerate(context_passages)
)
user_message = f"Context:\n{context}\n\nQuestion: {question}"
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
temperature=0.1,
max_tokens=512
)
return response.choices[0].message.content
if __name__ == "__main__":
print(ask_question("Which models support reasoning tasks?"))Example output:
Based on the provided text, the models suited for reasoning include DeepSeek R1 671B MoE for deep reasoning, Qwen 3 32B for multilingual reasoning and agent workflows, DeepSeek V4 Flash for near state-of-the-art open-source reasoning, and Kimi K2.6 for advanced reasoning and agentic coding.Wrap up and next steps
This agent rebuilds embeddings on every run. In production, cache the vectors to a local file or load them into a vector database so you only embed new documents. You can also swap Llama 3.3 70B for Kimi K2.6 on Oxlo.ai if you need vision support or longer context, or switch to Qwen 3 32B for multilingual documents. Because Oxlo.ai bills per request rather than per token, iterating with larger prompts and different models stays predictable. See the pricing page to compare flat request pricing against your current token-based bill.
Top comments (0)