Integrating LLM with Existing Database

#aiinfrastructure #oxlo #ai

Most production applications already run on structured data. The challenge is not replacing your database, but connecting it to an LLM without adding latency, cost, or fragility. Whether you are building retrieval-augmented generation over Postgres, generating SQL from natural language, or giving an agent direct read-access to your warehouse, the integration path is similar: send context to the model, get a structured action back, and execute it against your store.

Pattern 1: RAG Over Existing Structured Stores

The simplest integration is retrieval-augmented generation. Instead of fine-tuning on your schema, you embed table documentation, past queries, or business logic and retrieve the relevant chunks at runtime. Oxlo.ai provides embeddings via BGE-Large and E5-Large through a standard OpenAI-compatible endpoint, so you can generate vectors without changing your client code.

import os
import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

response = client.embeddings.create(
    model="bge-large",  # or e5-large via Oxlo.ai embeddings endpoint
    input="Customer orders table: id, user_id, total_cents, created_at"
)

embedding = response.data[0].embedding
# Store in pgvector, Redis, or your existing vector extension

After retrieval, you inject the relevant context into a chat prompt. Because Oxlo.ai does not cold-start popular models, that first request after idle time returns immediately, which matters for synchronous database lookups in user-facing apps.

Pattern 2: Natural Language to SQL with Schema Context

Text-to-SQL fails when the model lacks schema context. In practice, you must inject CREATE TABLE statements, column descriptions, and few-shot examples into the prompt. This makes prompts long, which is where token-based billing hurts. Oxlo.ai uses request-based pricing, so one flat cost per API request regardless of prompt length. For database integrations that require extensive DDL context, this makes costs predictable. See Oxlo.ai pricing for plan details.

import os
import json
import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

schema = """
CREATE TABLE orders (
  id SERIAL PRIMARY KEY,
  user_id INT NOT NULL,
  total_cents INT NOT NULL,
  status VARCHAR(20),
  created_at TIMESTAMP
);
"""

messages = [
    {"role": "system", "content": "You are a SQL expert. Respond with only valid SQL inside a JSON object with key 'query'."},
    {"role": "user", "content": f"Schema:\n{schema}\n\nQuestion: What is the average order value in the last 30 days?"}
]

# Models such as Llama 3.3 70B, DeepSeek R1 671B MoE, and Qwen 3 32B work well here
response = client.chat.completions.create(
    model="llama-3.3-70b",
    messages=messages,
    response_format={"type": "json_object"}
)

query = json.loads(response.choices[0].message.content)["query"]
# Execute query against your database connection

For code-heavy generation, you can also reach for Oxlo.ai code-specialized models such as Qwen 3 Coder 30B, DeepSeek Coder, or Oxlo.ai Coder Fast through the same chat/completions endpoint.

Pattern 3: Agentic Database Access with Tool Use

For complex operations, give the model tools it can call. Oxlo.ai supports function calling and multi-turn conversations across its chat models, including agent-focused options like Qwen 3 32B, GLM 5, and Minimax M2.5. You define a tool for executing read-only SQL, and the model decides when to invoke it.

import os
import json
import openai

client = openai.OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

tools = [{
    "type": "function",
    "function": {
        "name": "run_sql",
        "description": "Execute a read-only SQL query against the Postgres database",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {"type": "string", "description": "Valid SQL SELECT statement"}
            },
            "required": ["query"]
        }
    }
}]

response = client.chat.completions.create(
    model="qwen-3-32b",  # or GLM 5, Minimax M2.5 for agentic tool use
    messages=[{"role": "user", "content": "Show me revenue by month for active users"}],
    tools=tools,
    tool_choice="auto"
)

if response.choices[0].message.tool_calls:
    tool_call = response.choices[0].message.tool_calls[0]
    query = json.loads(tool_call.function.arguments)["query"]
    # Execute query, return results, and continue the conversation

Managing Cost and Latency with Long Context

Database schemas, documentation, and retrieved chunks quickly inflate prompt size. With token-based providers, longer inputs mean higher costs per request. Oxlo.ai is a developer-first AI inference platform with request-based pricing: one flat cost per API request regardless of prompt length. Unlike token-based providers such as Together AI, Fireworks AI, OpenRouter, Replicate, and Anyscale, cost does not scale with input length, so Oxlo.ai is significantly cheaper for long-context and agentic workloads that ship full schemas or conversation history with every call.

This pricing model removes the penalty for few-shot prompting and detailed schema descriptions. You can include CREATE TABLE statements, index hints, and example rows without worrying about token count. Oxlo.ai also offers no cold starts on popular models, so your database-backed assistant stays responsive.

Selecting the Right Model for Database Tasks

Oxlo.ai hosts 45+ open-source and proprietary models across 7 categories, fully OpenAI SDK compatible. For database integration, choose based on the task:

Schema reasoning and complex SQL: DeepSeek R1 671B MoE, DeepSeek V4 Flash, or Kimi K2.6.
General-purpose NL2SQL: Llama 3.3 70B or Qwen 3 32B.
Code-specific generation: Qwen 3 Coder 30B, DeepSeek Coder, or Oxlo.ai Coder Fast.
Embeddings for RAG: BGE-Large or E5-Large via the embeddings endpoint.

Getting Started

Oxlo.ai is a fully OpenAI API compatible drop-in replacement. Change your base URL to https://api.oxlo.ai/v1 and you can use the same Python, Node.js, or cURL patterns you already know. The Free plan includes 60 requests per day and access to 16+ free models, including DeepSeek V3.2 on the free tier, so you can prototype your database integration before upgrading. Paid plans start at Pro for 1,000 requests per day across all models, with Premium offering 5,000 requests per day and priority queue access. Enterprise plans provide custom unlimited volume with dedicated GPUs.

Because Oxlo.ai does not charge by the token, you can send full schema dumps and multi-turn conversation histories without surprise bills. For teams running agentic workloads or long-context RAG over large databases, request-based pricing can be 10-100x cheaper than token-based for long-context workloads. Check the details at https://oxlo.ai/pricing.

DEV Community