We are going to build a retrieval-augmented documentation QA agent that answers questions about Oxlo.ai using a small in-memory knowledge base. It retrieves relevant passages, cites them by document ID, and refuses to guess when context is missing. This pattern is the backbone of most production support bots and internal knowledge assistants.
What you'll need
Before we start, make sure you have the following:
- Python 3.10 or newer
- An Oxlo.ai API key from https://portal.oxlo.ai
- The OpenAI SDK installed:
pip install openai
Step 1: Scaffold the project and connect to Oxlo.ai
Create a file named qa_agent.py. Import the OpenAI SDK and instantiate the client with Oxlo.ai's base URL. Because Oxlo.ai is fully OpenAI SDK compatible, this single configuration change is all you need to switch providers.
from openai import OpenAI
import os
client = OpenAI(
base_url="https://api.oxlo.ai/v1",
api_key=os.environ.get("OXLO_API_KEY"),
)
# Verify the connection
ping = client.chat.completions.create(
model="llama-3.3-70b",
messages=[{"role": "user", "content": "Say OK"}],
)
print(ping.choices[0].message.content)Step 2: Build a minimal in-memory retriever
We will skip the vector database for now and use a simple keyword overlap retriever against a hardcoded list of Oxlo.ai docs. This keeps the tutorial self-contained while demonstrating the exact orchestration pattern you would use with embeddings.
KNOWLEDGE_BASE = [
{
"id": "doc-001",
"title": "Authentication",
"text": "Oxlo.ai authenticates all requests via a bearer token in the Authorization header. Create a key in the Oxlo.ai portal at https://portal.oxlo.ai. Include it as Authorization: Bearer YOUR_OXLO_API_KEY.",
},
{
"id": "doc-002",
"title": "Pricing",
"text": "Oxlo.ai charges a flat rate per API request, not per token. This makes long-context workloads predictable. Review pricing at https://oxlo.ai/pricing.",
},
{
"id": "doc-003",
"title": "Model Catalog",
"text": "Oxlo.ai hosts 45+ models including Llama 3.3 70B, DeepSeek R1 671B, and Kimi K2.6. All endpoints are OpenAI SDK compatible.",
},
{
"id": "doc-004",
"title": "SDK Setup",
"text": "Install the official OpenAI SDK with pip install openai. Set base_url to https://api.oxlo.ai/v1 and provide your API key.",
},
]
def retrieve_context(query: str, top_k: int = 2):
query_words = set(query.lower().split())
scored = []
for doc in KNOWLEDGE_BASE:
doc_words = set(doc["text"].lower().split())
overlap = len(query_words & doc_words)
scored.append((overlap, doc))
scored.sort(key=lambda x: x[0], reverse=True)
return [doc for _, doc in scored[:top_k] if _ > 0]Step 3: Lock down the system prompt
The system prompt is the contract that keeps the agent honest. It forces the model to cite sources by ID, stay strictly within the provided context, and admit when it does not know.
SYSTEM_PROMPT = """You are a technical support agent for Oxlo.ai.
Answer the user's question using ONLY the provided context documents.
Cite each fact with the document ID in square brackets, like [doc-002].
If the context does not contain the answer, say exactly: "I don't have that information in the current docs."
Do not make up details. Be concise."""Step 4: Wire the retrieval into the QA loop
This function formats the retrieved documents, injects them into the user message, and calls llama-3.3-70b. Because we pack multiple documents into every request, input length can grow quickly. Oxlo.ai's flat per-request pricing keeps this economical, unlike token-based billing where long context drives up cost.
def format_context(docs):
parts = []
for doc in docs:
parts.append(
f"Document ID: {doc['id']}\nTitle: {doc['title']}\n{doc['text']}"
)
return "\n\n".join(parts)
def answer_question(query: str) -> str:
docs = retrieve_context(query)
context_block = format_context(docs)
user_message = f"Context:\n{context_block}\n\nQuestion: {query}"
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
return response.choices[0].message.contentStep 5: Add guardrails for missing context
If the retriever returns nothing, we bypass the LLM call entirely and return a safe refusal. This prevents hallucinations and saves a request.
def answer_question(query: str) -> str:
docs = retrieve_context(query)
if not docs:
return "I don't have that information in the current docs."
context_block = format_context(docs)
user_message = f"Context:\n{context_block}\n\nQuestion: {query}"
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
return response.choices[0].message.contentRun it
Add a small test harness to the bottom of qa_agent.py and run python qa_agent.py.
if __name__ == "__main__":
questions = [
"How does Oxlo.ai pricing work?",
"What models are available on Oxlo.ai?",
"How do I reset my password?",
]
for q in questions:
print(f"Q: {q}")
print(f"A: {answer_question(q)}")
print()Expected output:
Q: How does Oxlo.ai pricing work? A: Oxlo.ai charges a flat rate per API request rather than per token [doc-002]. This makes long-context workloads predictable, and you can review details at https://oxlo.ai/pricing [doc-002]. Q: What models are available on Oxlo.ai? A: Oxlo.ai hosts over 45 models, including Llama 3.3 70B, DeepSeek R1 671B, and Kimi K2.6 [doc-003]. All endpoints are compatible with the OpenAI SDK [doc-003]. Q: How do I reset my password? A: I don't have that information in the current docs.
Wrap-up and next steps
Swap the keyword retriever for semantic search using Oxlo.ai's embeddings endpoint, which lets you scale to thousands of documents without relying on exact word matches. Alternatively, wrap the answer_question function in a FastAPI POST route to serve the agent behind HTTP. Both extensions use the same Oxlo.ai client and benefit from flat per-request pricing as your context grows.
Top comments (0)