Introduction
In this tutorial we will build a Document Research Agent that ingests a long technical document and answers citation-backed questions in a single API call. This demonstrates how an LLM inference engine handles real-world long-context workloads, and why Oxlo.ai's flat per-request pricing (see https://oxlo.ai/pricing) makes it practical to pass entire documents instead of managing chunks. If you are building RAG pipelines or agentic research tools, this pattern removes a lot of plumbing.
What you'll need
- An Oxlo.ai API key from https://portal.oxlo.ai
- Python 3.10 or newer
- The OpenAI SDK:
pip install openai
Step 1: Connect to Oxlo.ai
First, I verify that the client can reach Oxlo.ai and that my key is active. I use the general-purpose Llama 3.3 70B model for this smoke test because it starts instantly with no cold starts.
from openai import OpenAI
client = OpenAI(base_url="https://api.oxlo.ai/v1", api_key="YOUR_OXLO_API_KEY")
response = client.chat.completions.create(
model="llama-3.3-70b",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Say 'Inference engine is live' and nothing else."},
],
)
print(response.choices[0].message.content)Step 2: Load a long document
Next, I need a document large enough that token-based pricing would sting. I keep the text in a Python variable so the tutorial is self-contained, but in production this streams from a PDF parser or web scrape.
DOCUMENT = """
The Oxlo.ai Inference Platform
Oxlo.ai provides a developer-first AI inference platform with request-based pricing.
Instead of charging per token, Oxlo.ai charges one flat cost per API request regardless
of prompt length. This makes it significantly cheaper for long-context and agentic
workloads compared to token-based providers.
The platform hosts 45+ open-source and proprietary models across seven categories:
LLMs, code models, vision models, image generation, audio, embeddings, and object
detection. All endpoints are fully OpenAI SDK compatible, so switching from another
provider is a one-line change to the base URL.
Flagship models include Llama 3.3 70B for general-purpose tasks, DeepSeek R1 671B MoE
for deep reasoning, Kimi K2.6 for advanced reasoning and agentic coding with a 131K
context window, and DeepSeek V4 Flash which offers a 1 million token context and
efficient MoE architecture.
Because cost does not scale with input length, agents can afford to include full
conversation histories, large codebases, or entire research papers in every request.
There are no cold starts on popular models, so latency stays predictable.
"""Step 3: Define the agent prompt
The system prompt is the only part of the agent that never changes. I keep it strict: the model must cite specific sections and return valid JSON so downstream code can trust the shape of the output.
SYSTEM_PROMPT = """
You are a precise research assistant. Your job is to answer the user's question using
ONLY the provided document. Follow these rules exactly:
1. Quote the specific sentence or paragraph from the document that supports your answer.
2. If the document does not contain the answer, say "not found in source".
3. Return your response as a JSON object with exactly two keys:
- "answer": a concise answer string.
- "citation": the exact quoted text from the document, or null if not found.
Do not include markdown code fences in your output. Output raw JSON only.
"""Step 4: Build the agent
Now I wire the pieces together. I format the user message to include both the document and the question, then send the full payload to Kimi K2.6. Its 131K context window and reasoning capabilities handle long source text well, and because Oxlo.ai pricing is per request, the cost is the same whether I send two paragraphs or two hundred.
import json
def research_agent(question: str, document: str) -> dict:
user_message = f"""Document:
{document}
Question:
{question}"""
response = client.chat.completions.create(
model="kimi-k2.6",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
raw = response.choices[0].message.content
return json.loads(raw)Step 5: Enforce structured output
Raw string parsing is brittle. I switch on JSON mode so the model is constrained to valid JSON, and I set temperature to 0.1 to keep citations verbatim. This is the final agent code.
import json
def research_agent(question: str, document: str) -> dict:
user_message = f"""Document:
{document}
Question:
{question}"""
response = client.chat.completions.create(
model="kimi-k2.6",
temperature=0.1,
response_format={"type": "json_object"},
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": user_message},
],
)
raw = response.choices[0].message.content
return json.loads(raw)Run it
I call the agent with a question that requires scanning the entire document. The flat request pricing means I do not have to worry about the document length pushing me into a higher cost tier.
if __name__ == "__main__":
question = "Which model on Oxlo.ai supports a 1 million token context window?"
result = research_agent(question, DOCUMENT)
print(json.dumps(result, indent=2))Example output:
{
"answer": "DeepSeek V4 Flash supports a 1 million token context window.",
"citation": "DeepSeek V4 Flash which offers a 1 million token context and efficient MoE architecture."
}Wrap up
This agent shows how an LLM inference engine removes infrastructure work. Oxlo.ai handles model hosting, routing, and scaling, while the flat per-request pricing makes long-context agents economically viable.
Two concrete next steps. First, wrap the research_agent function in a FastAPI endpoint so other services can submit documents via HTTP. Second, add a validation layer using Oxlo.ai's BGE-Large embedding endpoint to verify that the citation text actually exists in the source document before returning it to the user.
Top comments (0)