Integrating LLM with Existing Database: A Step-by-Step Guide

Most production applications already run on structured data in PostgreSQL, MySQL, or SQL Server. Adding an LLM layer does not require migrating that data to a new vector store. Instead, you can connect an inference API directly to your existing database, using the model to translate natural language into safe queries or to augment retrieval pipelines. This guide shows a concrete, code-first path to doing that, using Oxlo.ai as the inference backend.

Architecture Overview

Before writing code, map out how the LLM interacts with your database. There are two dominant patterns that keep your existing data in place: direct text-to-SQL generation, and retrieval-augmented generation over schema metadata. In the text-to-SQL pattern, the model receives your schema and a user question, then returns a query you validate and execute. In the RAG pattern, you embed table descriptions and documentation, retrieve the most relevant fragments, and feed those into the prompt to reduce noise. Both approaches work well with Oxlo.ai because the platform exposes chat, embedding, and code models through a single OpenAI-compatible endpoint.

Choosing Your Integration Pattern

Select a pattern based on schema complexity and user intent. Text-to-SQL is ideal when your schema is stable and fits comfortably inside the context window. If you manage hundreds of tables, a schema RAG layer is safer. You can also combine them: use embeddings to retrieve the right tables, then ask a coding model such as DeepSeek Coder on Oxlo.ai to write the SQL. Oxlo.ai carries both the embedding models, BGE-Large and E5-Large, and the chat models needed for either path, so you do not need to split traffic across providers.

Preparing Your Schema and Metadata

The quality of generated SQL depends on the context you provide. Export concise CREATE TABLE statements, primary and foreign keys, and a few representative rows. Avoid dumping the entire database if it exceeds the model's context limit. For sensitive columns, annotate constraints without including PII. Store this metadata in a file or in a dedicated governance table so your application can hydrate prompts consistently.

Implementing Text-to-SQL with Oxlo.ai

Because Oxlo.ai is fully OpenAI SDK compatible, you can drop it into an existing Python service with a two-line change. Set the base URL to https://api.oxlo.ai/v1 and choose a model suited to code generation. DeepSeek Coder and Qwen 3 Coder 30B are strong candidates for SQL, while Llama 3.3 70B handles general-purpose schema reasoning.

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.oxlo.ai/v1",
    api_key=os.environ["OXLO_API_KEY"]
)

SCHEMA = """
CREATE TABLE orders (
    order_id INT PRIMARY KEY,
    customer_id INT REFERENCES customers(customer_id),
    order_date DATE,
    total DECIMAL(10,2)
);
"""

def generate_sql(question: str) -> str:
    resp = client.chat.completions.create(
        model="llama-3.3-70b",  # or the Oxlo.ai identifier for DeepSeek Coder
        messages=[
            {"role": "system", "content": f"You write safe, read-only SQL.\n\n{SCHEMA}"},
            {"role": "user", "content": question}
        ],
        temperature=0.1,
        max_tokens=500
    )
    return resp.choices[0].message.content

if __name__ == "__main__":
    print(generate_sql("What is the average order total by month?"))

The example above sends the full schema in every request. Because Oxlo.ai uses request-based pricing, you pay a flat rate per call no matter how long the schema description is. That makes this simple pattern economical compared to token-based providers, where large prompts incur large input charges. See the Oxlo.ai