Revolutionizing Information Retrieval with LLMs: Oxlo's Approach

#aiinfrastructure #oxlo #ai

Retrieval systems have moved past simple keyword matching. Modern information retrieval now depends on dense vector search, re-ranking, and generative summarization powered by large language models. These pipelines embed documents into semantic representations, retrieve relevant chunks, and synthesize answers through long-context inference or multi-step agentic reasoning. For engineering teams, the challenge is no longer whether LLMs can improve retrieval accuracy, but how to run these workloads without costs that scale unpredictably with every additional source document.

The Economics of Long-Context Retrieval

Token-based pricing creates a structural penalty on retrieval systems that feed long documents or large result sets into a model. Every retrieved chunk, every page of context, and every re-ranking pass adds input tokens. For agentic retrieval, where a model iteratively searches, filters, and synthesizes across dozens of sources, these costs compound rapidly.

Oxlo.ai uses request-based pricing. Each API call carries one flat cost regardless of prompt length. For retrieval pipelines that pass entire PDFs, codebases, or conversation histories into the context window, this can be 10-100x cheaper than token-based alternatives. Instead of trimming chunks to save tokens, you can send the full document and let the model reason over it. You can explore the pricing details at https://oxlo.ai/pricing.

Building a Two-Stage Retrieval Pipeline

A production retrieval system usually combines embedding-based search with generative answer synthesis. Oxlo.ai provides both endpoints through a fully OpenAI-compatible API, so you can drop the SDK into existing code by changing the base URL.

import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key="YOUR_OXLO_API_KEY"
)

# Stage 1: Embed the query and documents with BGE-Large
docs = [
    "Oxlo.ai offers flat per-request pricing for LLM inference.",
    "Token-based providers scale cost with input length.",
    "Embeddings convert text into dense vectors for semantic search."
]

embed_response = client.embeddings.create(
    model="bge-large",
    input=[docs[0]] + docs[1:]
)

# Compute cosine similarity and pick top-k chunks
# (similarity logic omitted for brevity)
top_chunks = [docs[0], docs[2]]

# Stage 2: Synthesize an answer with structured JSON output
completion = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=[
        {"role": "system", "content": "Answer using the provided context. Respond in JSON with keys 'answer' and 'sources'."},
        {"role": "user", "content": f"Context: {' '.join(top_chunks)}\n\nQuestion: How does Oxlo.ai pricing work?"}
    ],
    response_format={"type": "json_object"}
)

print(completion.choices[0].message.content)

This pattern works with any Oxlo.ai chat model. For reasoning-heavy retrieval, you can swap llama-3.3-70b for deepseek-r1-671b or qwen-3-32b. For extremely long documents, kimi-k2.6 with its 131K context window or deepseek-v4-flash with 1M context can ingest entire reports in a single request.

When to Embed, When to Generate

Embedding retrieval remains the right choice for large-scale filtering over millions of records. Oxlo.ai hosts bge-large and e5-large for this exact purpose. Once you have a candidate set, however, modern LLMs can act as powerful re-rankers and readers. A model like deepseek-r1-671b or kimi-k2-thinking can evaluate whether a retrieved passage truly answers the query, detect contradictions, and perform chain-of-thought verification before returning a result.

Because Oxlo.ai charges per request rather than per token, you can afford to run these verification steps without watching metered costs climb. You can even implement agentic loops where glm-5 or minimax-m2.5 iteratively calls search tools, evaluates evidence, and refines answers. No cold starts on popular models means these interactive retrieval agents remain responsive.

Multimodal Documents and Agentic Verification

Information retrieval is no longer limited to plain text. Engineering teams need to extract data from slide decks, scanned PDFs, and diagrams. Oxlo.ai offers vision models such as gemma-3-27b and kimi-vl-a3b that accept image inputs through the standard chat completions endpoint. You can pass a screenshot of a technical diagram alongside a text query and receive a structured description or answer.

For autonomous retrieval systems, agentic models like glm-5 and minimax-m2.5 support function calling and tool use. A retrieval agent can decide to query a vector database, parse a PDF page as an image, and cross-reference results, all within a single conversation loop. Because Oxlo.ai is fully OpenAI SDK compatible, migrating an existing agent from another provider requires only a base URL change.

Getting Started

Oxlo.ai offers a free tier with 60 requests per day across 16+ models, including a 7-day full-access trial. This is enough to prototype a retrieval pipeline, test embedding quality, and benchmark long-context accuracy against your current setup. If you are running retrieval workloads on token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, or Anyscale, the flat request model removes the cost penalty that usually forces aggressive chunking and truncated context windows.

Set your base URL to https://api.oxlo.ai/v1 and use the standard OpenAI SDK patterns for embeddings, chat completions, JSON mode, and function calling. For teams moving production retrieval systems, the Enterprise plan offers dedicated GPUs and guaranteed savings against existing providers.