RAG Agents Tutorial

Retrieval-Augmented Generation (RAG) agents combine vector search with autonomous tool use to ground large language models in private data and external APIs. Unlike simple Q&A bots, a RAG agent can iterate: it plans a query, retrieves documents, critiques its own evidence, and decides whether to search again or respond. This loop produces more accurate answers, but it also multiplies the number of LLM calls and the total tokens consumed. For teams running agentic workloads at scale, inference cost and latency become architectural constraints, not just line items.

Architecture Overview

A production RAG agent is not a single prompt. It is a state machine with three distinct layers. The retrieval layer embeds queries and searches a vector store. The reasoning layer synthesizes evidence, detects gaps, and decides whether to call a tool. The generation layer produces the final answer with citations.

Oxlo.ai supports this stack end to end. You can generate embeddings through the /embeddings endpoint using BGE-Large or E5-Large, run reasoning with tool-enabled models such as Llama 3.3 70B or Qwen 3 32B, and stream the final response back to the user. Because the platform is fully OpenAI SDK compatible, you can prototype with your existing Python tooling and switch the base URL to https://api.oxlo.ai/v1.

Setting Up the Retrieval Pipeline

Start by chunking your documents and indexing them with dense embeddings. The quality of retrieval determines the ceiling of your agent, so use a strong embedding model.

import openai
import numpy as np

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

def get_embeddings(texts):
    response = client.embeddings.create(
        model="bge-large",
        input=texts
    )
    return [item.embedding for item in response.data]

# Example: embed document chunks
chunks = [
    "Oxlo.ai offers flat per-request pricing for open-source LLMs.",
    "RAG agents require iterative retrieval and reasoning loops."
]
vectors = get_embeddings(chunks)

Store these vectors in any vector database. When a user asks a question, embed the query with the same Oxlo.ai endpoint and run a similarity search. Keep the top-k chunks and their metadata so the agent can cite sources later.

Building the Agent Loop

The agent is a loop that repeats until it decides to answer or hits a max-step limit. Each iteration sends the current state, available tools, and retrieved context to the LLM. Oxlo.ai models support function calling and JSON mode, so you can define a search_knowledge_base tool and let the model choose when to invoke it.

import json

from typing import List, Dict