Building and Debugging RAG Agents

#aiinfrastructure #oxlo #ai

Retrieval-augmented generation (RAG) agents combine vector search with tool-augmented language models to answer questions grounded in private data. Moving from a notebook prototype to a reliable production agent requires rigorous control over context assembly, reasoning loops, and failure modes. This guide walks through a practical architecture, implementation patterns, and debugging techniques that keep latency low and responses accurate. We will use Oxlo.ai as the inference backend because its request-based pricing and broad model catalog simplify cost management once retrieved documents start filling the context window.

RAG Agent Architecture

A production RAG agent typically has four layers: an embedding and retrieval stage, a tool schema that exposes search to the LLM, a reasoning loop that decides when to retrieve, and a final synthesis step that grounds the answer in the returned chunks. Keeping these layers distinct makes debugging easier because you can test retrieval quality in isolation before you ever invoke a generator.

Embedding and retrieval: Chunk documents, encode them with an embedding model, and index them in a vector store.
Tool schema: Describe the search function to the LLM so it can request relevant context.
Reasoning loop: Handle tool calls, inject retrieved chunks into the conversation history, and ask the model to reason again.
Answer synthesis: Generate the final response with citations tied to the retrieved chunks.

Retrieval and Embedding Setup

Start by turning documents into vectors. Oxlo.ai hosts BGE-Large and E5-Large embedding models that are fully compatible with the OpenAI SDK. The snippet below generates an embedding you can store in pgvector, ChromaDB, or any other vector store.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

response = client.embeddings.create(
    model="bge-large",
    input=["Oxlo.ai offers flat per-request pricing for long-context workloads."]
)
embedding = response.data[0].embedding

Chunk sizing and overlap matter more than the embedding model itself. A good starting point is 512 tokens with 64 tokens of overlap, then adjust based on the structure of your documents. Once the vectors are indexed, build a simple search function that returns the top-k chunks plus metadata such as source URLs.

Agent Loop and Tool Use

With retrieval in place, expose it as a tool via the chat completions API. The model can then decide whether to answer from memory or call the search function. After the model requests a tool call, execute the search locally, append the results as a new message with role tool, and send the updated conversation back for the final answer. Models such as Qwen 3 32B and GLM 5 on Oxlo.ai handle multi-turn tool interactions reliably.

tools = [
    {
        "type": "function",
        "function": {
            "name": "search_knowledge_base",
            "description": "Retrieve relevant chunks from internal docs",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "top_k": {"type": "integer", "default": 5}
                },
                "required": ["query"]
            }
        }
    }
]

messages = [
    {"role": "system", "content": "You are a helpful assistant with access to internal docs."},
    {"role": "user", "content": "How does request-based pricing work?"}
]

response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=messages,
    tools=tools,
    tool_choice="auto"
)

If the response contains a tool_calls field, run the search, format the chunks into a string, and append a tool message to messages before calling the API again. For agents that need structured output, you can also enable JSON mode to force the model to return a JSON object instead of free text.

Debugging Techniques

RAG failures usually stem from retrieval, context overflow, or reasoning. Use the following practices to isolate them.

Trace context assembly. Log the exact text inserted into the prompt after retrieval. Small formatting changes, such as adding document separators or citation markers, can alter accuracy significantly.

Evaluate retrieval separately. Measure hit rate and mean reciprocal rank on a labeled dataset before you ever call the LLM. If the top chunks are irrelevant, no generator will rescue the answer.

Stream and inspect. Enable streaming responses so you can abort early when the model drifts. Oxlo.ai supports streaming on chat completions, which lets you surface partial reasoning in real time.

stream = client.chat.completions.create(
    model="deepseek-r1-671b",
    messages=messages,
    tools=tools,
    stream=True
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Swap models quickly. Oxlo.ai offers more than 45 models under one API key. If a large reasoning model such as DeepSeek R1 671B MoE is overkill for a simple lookup, switch to a faster generalist such as Llama 3.3 70B or DeepSeek V4 Flash without rewriting client code. This makes A/B testing straightforward.

Cost and Latency Optimization with Oxlo.ai

RAG workloads often balloon in cost because retrieved documents lengthen every prompt. On token-based providers, each additional chunk raises the bill. Oxlo.ai uses request-based pricing: one flat cost per API call regardless of how many tokens are in the prompt. For agents that retrieve ten pages of documentation per turn, this can yield substantial savings. See the Oxlo.ai pricing page for current plan details.

Other Oxlo.ai features that matter for agents include:

No cold starts on popular models, so your agent loop stays responsive.
Native function calling and JSON mode across flagship models, which lets you enforce structured tool arguments.
A broad model catalog spanning embeddings (BGE-Large, E5-Large), reasoning specialists (DeepSeek R1, Kimi K2.6), and efficient Mixture-of-Experts (DeepSeek V4 Flash, GLM 5) so you can match model capability to task complexity.
Fully OpenAI SDK compatibility means you can point existing RAG frameworks such as LangChain or LlamaIndex to https://api.oxlo.ai/v1 with a single line change.

If you are prototyping, the Oxlo.ai free tier includes 60 requests per day across more than 16 models, which is enough to iterate on retrieval strategies before committing to a paid plan.

Building robust RAG agents is as much about observability and cost control as it is about model selection. By instrumenting the retrieval boundary, streaming outputs for early inspection, and hosting inference on Oxlo.ai, you keep long-context agent loops predictable and affordable.

DEV Community