DEV Community: Lakhan Malviya

Your Schema Isn't Enough: Build a SQL Agent That Reads Business Knowledge First (OKF Series, Part 4)

Lakhan Malviya — Sun, 04 Oct 2026 16:29:29 +0000

Ask a data team how they calculate their on-time delivery rate, and the answer rarely comes from the database. The columns are all there: promised dates, delivery dates, order status. The rule that turns them into one number lives somewhere else, in a wiki page, a dashboard, or a senior analyst's head.

SQL agents run into this wall all the time. Here's a real example from LiveSQLBench, a public benchmark for agents that turn questions into SQL. One of its databases tracks disaster relief, and one of its questions asks:

I need to analyze all distribution hubs based on their Resource Utilization Ratio. Please show the hub registry ID, the calculated RUR value, and their Resource Utilization Classification. Sort the results by RUR from highest to lowest.

The database has a distributionhubs table, and three of its columns are clearly involved (only the relevant columns are shown):

distributionhubs (hubregistry, hubutilpct, storecapm3, storeavailm3, ...)

But no column is called "Resource Utilization Ratio", and nothing in the schema says how to combine these three. Nothing says where "High Utilization" starts, either. Both answers live in a separate file of business rules:

RUR = (hubutilpct / 100) × (storecapm3 / (storeavailm3 + 1))

High Utilization       RUR > 5
Moderate Utilization   2 ≤ RUR ≤ 5
Low Utilization        RUR < 2

An agent that sees only the schema has to invent the formula and the thresholds. Whatever it invents, the query runs without an error and returns a neat table. The answer is wrong, and nothing in the output tells you it was a guess.

So the schema isn't enough. The agent also needs the business knowledge, and it needs to find the right piece without reading every rule for every question.

In this part, we build a SQL agent that looks up business knowledge before it writes a query. The knowledge lives in an OKF bundle: a folder of small markdown files, one per table and one per business rule, with an index file in each folder. The agent starts at the top index, opens only the files the question needs, and then queries the database. By the end, you'll have:

One terminal command that answers a question about a real PostgreSQL database.
A read-only database user, so the SQL a model writes can't change or damage your data.
A trace for every answer: which files the agent opened, every SQL attempt and error, and the tokens and cost it took.
A side-by-side run of the same question with the bundle and with all the raw docs pasted into the prompt.

We use free open models hosted on NVIDIA's API. Switching to Ollama, Groq, OpenAI, Gemini or Anthropic is a change in the config file.

If you followed the previous part, the bundle you compiled there is exactly what this agent reads. If you're starting here, the tutorial guide builds all the bundles with one command.

Set up the project for the agent

We add a second package, okf_agent, next to the compiler, and make the whole project installable. Installing it is what gives us the okf-agent terminal command at the end of this post.

Here's where we're heading. Each file is written, in full, in the step that needs it:

okf-sql-knowledge/
├── data/
│   └── livesqlbench-base-lite/
├── bundles/
│   └── compiled/            the OKF bundles from the compiler
├── okf_compiler/            the compiler
├── okf_agent/
│   ├── __init__.py
│   ├── config.py           read the settings file
│   ├── llm.py              call the model through LiteLLM
│   ├── tools.py            the tools the agent can call
│   ├── agent.py            the agent loop
│   ├── trace.py            record every step of a run
│   ├── database.py         run SQL safely
│   ├── prepare.py          start the database, create the read-only user
│   └── cli.py              the okf-agent command
├── runs/                    one JSONL line per question asked
├── config.yaml              which model to use, and other settings
├── .env                     API key and passwords (never committed)
├── .env.example             the template for .env
├── docker-compose.yml       the benchmark database
├── PROVIDERS.md             how to use each model provider
├── pyproject.toml
└── README.md

Create the okf_agent/ folder with an empty __init__.py inside it. Then install the project as described in the tutorial guide. It adds three dependencies: LiteLLM, our one connection to every model provider; psycopg, which connects Python to PostgreSQL; and python-dotenv, which reads the API key and passwords from a .env file.

Connect to a model and get a test answer

An agent is a model that can ask for tools: "open this file", "run this query". Not every model can, and the ones that can don't all report what each call cost. So before we build anything, we check that our model can do both.

We use free open models hosted on NVIDIA's API. Create a free key there and add it to your .env file as NVIDIA_NIM_API_KEY, as shown in the provider guide. The same guide shows how to use Ollama, Groq, OpenAI, Gemini or Anthropic instead.

The settings file

Create config.yaml in the project root:

model:
  name: nvidia_nim/nvidia/nemotron-3-super-120b-a12b
  temperature: 0.0
  max_tokens: 4096
  callbacks: []   # optional trace UI: [langfuse] or [langsmith]

agent:
  bundles: bundles/compiled
  raw_docs: data/livesqlbench-base-lite
  max_steps: 15
  runs: runs

database:
  host: localhost
  port: 5432
  agent_user: agent_ro
  statement_timeout_s: 30
  max_rows: 20

The model name has two parts: the provider (nvidia_nim) and the model's ID at that provider. To switch provider or model, you change this one line. callbacks can also send every model call to a trace UI such as Langfuse or LangSmith. It's off by default, and the provider guide shows how to turn it on. The agent and database settings are used in later steps.

Notice what's not in this file: the API key and the database passwords. config.yaml is committed to git and gets shared, so every secret lives in .env, which git ignores (see the tutorial guide).

Create okf_agent/config.py to read both:

"""Read the settings file, and the secrets from the environment."""

import os
from dataclasses import dataclass
from pathlib import Path

import yaml
from dotenv import load_dotenv


@dataclass
class ModelConfig:
    name: str
    temperature: float
    max_tokens: int
    callbacks: list[str]


@dataclass
class AgentConfig:
    bundles: Path
    raw_docs: Path
    max_steps: int
    runs: Path


@dataclass
class DatabaseConfig:
    host: str
    port: int
    agent_user: str
    statement_timeout_s: int
    max_rows: int

    @property
    def admin_user(self) -> str:
        return secret("POSTGRES_USER")

    @property
    def admin_password(self) -> str:
        return secret("POSTGRES_PASSWORD")

    @property
    def agent_password(self) -> str:
        return secret("AGENT_DB_PASSWORD")


@dataclass
class Config:
    model: ModelConfig
    agent: AgentConfig
    database: DatabaseConfig


def secret(name: str) -> str:
    value = os.environ.get(name)
    if not value:
        raise SystemExit(f"{name} is not set. Add it to your .env file (see .env.example).")
    return value


def load_config(path: str | Path = "config.yaml") -> Config:
    load_dotenv()  # read .env into the environment
    raw = yaml.safe_load(Path(path).read_text(encoding="utf-8"))
    agent = raw["agent"]
    return Config(
        model=ModelConfig(**raw["model"]),
        agent=AgentConfig(
            bundles=Path(agent["bundles"]),
            raw_docs=Path(agent["raw_docs"]),
            max_steps=agent["max_steps"],
            runs=Path(agent["runs"]),
        ),
        database=DatabaseConfig(**raw["database"]),
    )

One function for every model call

Create okf_agent/llm.py. Every call the agent makes to the model goes through chat(), so this is the one place where we measure tokens, cost and time:

"""Call the model through LiteLLM, and measure every call."""

import json
import time
from dataclasses import dataclass, field

import litellm

from okf_agent.config import ModelConfig

litellm.drop_params = True  # skip settings a provider doesn't support


@dataclass
class Reply:
    message: dict
    tool_calls: list[dict] = field(default_factory=list)
    input_tokens: int = 0
    output_tokens: int = 0
    reasoning_tokens: int = 0
    cost_usd: float | None = None
    seconds: float = 0.0
    waited_seconds: float = 0.0


def chat(model: ModelConfig, messages: list[dict], tools: list[dict] | None = None,
         max_retries: int = 5) -> Reply:
    litellm.success_callback = model.callbacks
    waited = 0.0
    for attempt in range(max_retries + 1):
        start = time.perf_counter()
        try:
            response = litellm.completion(
                model=model.name,
                messages=messages,
                tools=tools or None,
                temperature=model.temperature,
                max_tokens=model.max_tokens,
            )
            break
        except litellm.RateLimitError:
            if attempt == max_retries:
                raise
            pause = 5 * 2**attempt
            time.sleep(pause)
            waited += pause
    seconds = time.perf_counter() - start

    message = response.choices[0].message
    tool_calls = [
        {"id": c.id, "name": c.function.name, "arguments": json.loads(c.function.arguments or "{}")}
        for c in (message.tool_calls or [])
    ]
    usage = response.usage
    details = getattr(usage, "completion_tokens_details", None)
    try:
        cost = litellm.completion_cost(completion_response=response)
    except Exception:
        cost = None  # no price known for this model

    return Reply(
        message=message.model_dump(exclude_none=True),
        tool_calls=tool_calls,
        input_tokens=usage.prompt_tokens,
        output_tokens=usage.completion_tokens,
        reasoning_tokens=getattr(details, "reasoning_tokens", 0) or 0,
        cost_usd=cost,
        seconds=round(seconds, 2),
        waited_seconds=waited,
    )


if __name__ == "__main__":
    from okf_agent.config import load_config

    config = load_config()
    test_tool = {
        "type": "function",
        "function": {
            "name": "count_rows",
            "description": "Count the rows in a database table.",
            "parameters": {
                "type": "object",
                "properties": {"table": {"type": "string"}},
                "required": ["table"],
            },
        },
    }
    question = [{"role": "user", "content": "How many rows are in the distributionhubs table?"}]
    reply = chat(config.model, question, tools=[test_tool])

    print(f"model:      {config.model.name}")
    print(f"tool calls: {[(c['name'], c['arguments']) for c in reply.tool_calls] or 'none'}")
    print(f"text:       {reply.message.get('content') or '-'}")
    print(f"tokens:     {reply.input_tokens} in, {reply.output_tokens} out "
          f"({reply.reasoning_tokens} reasoning)")
    print(f"cost:       {'unknown' if reply.cost_usd is None else f'${reply.cost_usd:.6f}'}")
    print(f"time:       {reply.seconds}s")

Notice three things about what chat() measures:

Token counts come from the provider's response, not from an estimate. That's how we'll measure what each answer really costs.
Reasoning tokens are counted separately. Some models "think" before they answer, and those tokens are billed too, so a reasoning model can use many more tokens for the same answer.
Time spent waiting for a rate limit is kept apart from the call itself. NVIDIA's free tier limits how many requests you can send per minute. When we hit that limit, we pause and retry, and the pause shouldn't count as the model being slow.

Run the test

The block at the bottom of llm.py offers the model one made-up tool, count_rows, and asks how many rows a table has. Nothing gets executed. We only check that the model asks for the tool instead of guessing a number:

uv run -m okf_agent.llm

Output:

model:      nvidia_nim/nvidia/nemotron-3-super-120b-a12b
tool calls: [('count_rows', {'table': 'distributionhubs'})]
text:       -
tokens:     279 in, 91 out (62 reasoning)
cost:       unknown
time:       1.6s

The model asked for count_rows with the table name, which is exactly what our agent needs: it will ask for files and queries the same way. If tool calls says none, the model can't call tools; pick another model in config.yaml.

Notice the reasoning tokens: 62 of the 91 output tokens were spent thinking before the model asked for the tool. A model that thinks first can use several times more tokens for the same answer, which is why we count them separately.

The cost shows as unknown because LiteLLM has no price for a free model. With a paid provider, the same line shows the price of the call.

Build the tools: read a bundle file, submit an answer

The model can't open a file or run a query itself. It can only ask for a tool by name, and our code runs it and sends back the result. So every tool has two parts: a description the model reads, and the Python that runs when the model asks for it.

We start with two tools. read_file reads one file from the bundle. submit_answer is how the agent hands in its final SQL and answer. The tool that runs SQL comes later, once the database is running.

Create okf_agent/tools.py:

"""The tools the agent can call: what the model is told about them, and the code behind them."""

import sys
from pathlib import Path

import yaml

READ_FILE = {
    "type": "function",
    "function": {
        "name": "read_file",
        "description": (
            "Read one file from the knowledge bundle and return its text. The path is relative "
            "to the bundle root, for example 'index.md' or 'tables/distributionhubs.md'."
        ),
        "parameters": {
            "type": "object",
            "properties": {"path": {"type": "string", "description": "Path of the file to read."}},
            "required": ["path"],
        },
    },
}

SUBMIT_ANSWER = {
    "type": "function",
    "function": {
        "name": "submit_answer",
        "description": (
            "Submit your final answer. Call it once, at the end, with the final SQL query "
            "and a short answer in plain language. If the request can't be answered with a "
            "SQL query over this database, leave sql empty and explain why in answer."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "sql": {"type": "string", "description": "The final PostgreSQL query, or ''."},
                "answer": {"type": "string", "description": "A short answer in plain language."},
            },
            "required": ["sql", "answer"],
        },
    },
}


class Toolbox:
    """The tools for one run. Without a bundle, the agent gets no read_file."""

    def __init__(self, bundle: Path | None):
        self.bundle = bundle.resolve() if bundle else None
        self.answer: dict | None = None

    def schemas(self) -> list[dict]:
        return [READ_FILE, SUBMIT_ANSWER] if self.bundle else [SUBMIT_ANSWER]

    def run(self, name: str, arguments: dict) -> str:
        handlers = {"submit_answer": self.submit_answer}
        if self.bundle:
            handlers["read_file"] = self.read_file
        if name not in handlers:
            return f"Error: there is no tool called {name}."
        try:
            return handlers[name](**arguments)
        except TypeError as error:
            return f"Error: wrong arguments for {name}: {error}"

    def read_file(self, path: str) -> str:
        target = (self.bundle / path.lstrip("/")).resolve()
        if not target.is_relative_to(self.bundle):
            return f"Error: {path} is not a file in the bundle."
        if target.is_dir():
            target = target / "index.md"
        if target.suffix != ".md":
            return f"Error: {path} is not a file in the bundle."
        if target.is_file():
            return target.read_text(encoding="utf-8")
        if target.name == "index.md" and target.parent.is_dir():
            return self.list_folder(target.parent)  # OKF index files are optional
        return f"Error: {path} does not exist. Check the folder's index.md for the right name."

    def list_folder(self, folder: Path) -> str:
        """Build an index from the frontmatter, for a folder that has no index.md."""
        name = "the bundle root" if folder == self.bundle else folder.relative_to(self.bundle).as_posix()
        lines = [f"(No index.md here. Files in {name}:)"]
        for item in sorted(folder.iterdir()):
            if item.is_dir():
                lines.append(f"* [{item.name}/]({item.name}/index.md) - folder")
            elif item.suffix == ".md":
                text = item.read_text(encoding="utf-8")
                meta = yaml.safe_load(text.split("---", 2)[1]) if text.startswith("---") else {}
                title = (meta or {}).get("title", item.stem)
                lines.append(f"* [{title}]({item.name}) - {(meta or {}).get('description', '')}")
        return "\n".join(lines)

    def submit_answer(self, sql: str, answer: str) -> str:
        self.answer = {"sql": sql, "answer": answer}
        return "Answer received."


if __name__ == "__main__":
    from okf_agent.config import load_config

    database, path = sys.argv[1], sys.argv[2]
    toolbox = Toolbox(load_config().agent.bundles / database)
    print(toolbox.run("read_file", {"path": path}))

Notice five things:

Each description says only what the tool does and what to pass it. How to find your way around the bundle (start at index.md, follow the links) is not the tool's job. It goes in the agent's instructions, which we write in the next step.
read_file can only read markdown files inside this database's bundle. The path is resolved first, so a path like ../../.env can't climb out of the bundle folder.
OKF makes index files optional, so read_file copes without them. If a folder has no index.md, asking for it returns a list built from each file's frontmatter: the same title and one-line description an index would show. The agent still sees one folder at a time.
Errors go back to the model as text, not as Python exceptions. If the model asks for a file that doesn't exist, it reads the error and can try again, instead of the run crashing.
Without a bundle, there is no read_file. Later we ask the same question without the bundle, and the same Toolbox gives the agent only the other tools.

submit_answer only stores the answer. An empty sql means the agent decided the request can't be answered with a query. Ending the run is the agent loop's job, which we write next.

Try the tool yourself

The block at the bottom of tools.py runs read_file the same way the agent will. Read the bundle's front door first:

uv run -m okf_agent.tools disaster index.md

Output:

---
okf_version: "0.2"
---

# disaster database

* [Tables](tables/index.md) - PostgreSQL tables, with column meanings, JSON fields and joins.
* [Knowledge](knowledge/index.md) - Business rules: calculations, definitions and value illustrations.

Now try to leave the bundle:

uv run -m okf_agent.tools disaster ../../.env

Output:

Error: ../../.env is not a file in the bundle.

The tool refuses, and says so in a sentence the model can read. Whatever path a model makes up, read_file only ever returns knowledge from the bundle.

Build the agent loop

An agent is a loop. We send the model the question and the tools. If it asks for a tool, we run it, add the result to the conversation, and ask the model again. We stop when it calls submit_answer, or when it has used up its steps.

There's no database yet, so the agent can't run its SQL. It reads the bundle, writes a query and hands it in. That's enough to see whether it finds the right knowledge.

Create okf_agent/agent.py:

"""The agent loop: ask the model, run the tools it asks for, repeat until it submits."""

import json
import sys
from collections.abc import Callable
from dataclasses import dataclass, field
from datetime import datetime
from pathlib import Path

from okf_agent.config import Config, load_config
from okf_agent.llm import Reply, chat
from okf_agent.tools import Toolbox

INSTRUCTIONS = """You answer questions about the `{database}` PostgreSQL database by writing one SQL query.

Business terms in the questions, such as ratios, scores and classifications, have exact definitions: formulas, thresholds and the columns they use. Never guess a definition.

Only answer questions about this database's data. If the request is anything else, such as a greeting or a general request for help, call submit_answer right away with an empty sql and an answer that says what you can do. If you find that the database doesn't hold the data a question needs, do the same and say what's missing.

{knowledge}

When you know the query, call submit_answer with the final SQL and a one-sentence answer."""

BUNDLE_KNOWLEDGE = """The database's documentation is a knowledge bundle that you read with read_file.
1. Start with index.md, then read the index of the folder you need. Links in an index are relative to its folder; links that start with '/' are relative to the bundle root.
2. Open only the files the question needs: the business rules it names, the rules those depend on, and the tables whose columns they use.
3. Write the query with the exact formulas, thresholds, table names and column names you read."""

RAW_KNOWLEDGE = """The database's documentation is below: the schema, the meaning of each column, and the business rules. Write the query with the exact formulas, thresholds, table names and column names they give.

{docs}"""

SQL_NOTE = "\n\nBefore you submit, run your query with run_sql and fix any errors."


@dataclass
class ToolCall:
    name: str
    arguments: dict
    result: str


@dataclass
class Step:
    reply: Reply
    calls: list[ToolCall] = field(default_factory=list)


@dataclass
class Run:
    question: str
    database: str
    mode: str
    model: str
    started_at: str
    steps: list[Step] = field(default_factory=list)
    answer: dict | None = None
    stop_reason: str = ""
    error: str = ""


def raw_docs(folder: Path, database: str) -> str:
    parts = []
    for suffix in ("schema.txt", "column_meaning_base.json", "kb.jsonl"):
        path = folder / database / f"{database}_{suffix}"
        parts.append(f"## {path.name}\n\n{path.read_text(encoding='utf-8')}")
    return "\n\n".join(parts)


def system_prompt(config: Config, database: str, mode: str, toolbox: Toolbox) -> str:
    if mode == "bundle":
        knowledge = BUNDLE_KNOWLEDGE
    else:
        knowledge = RAW_KNOWLEDGE.format(docs=raw_docs(config.agent.raw_docs, database))
    names = [tool["function"]["name"] for tool in toolbox.schemas()]
    if "run_sql" in names:
        knowledge += SQL_NOTE
    return INSTRUCTIONS.format(database=database, knowledge=knowledge)


def assistant_message(reply: Reply) -> dict:
    """The model's turn, in the shape every provider accepts back."""
    message = {"role": "assistant", "content": reply.message.get("content") or ""}
    if reply.tool_calls:
        message["tool_calls"] = [
            {"id": c["id"], "type": "function",
             "function": {"name": c["name"], "arguments": json.dumps(c["arguments"])}}
            for c in reply.tool_calls
        ]
    return message


def ask(config: Config, database: str, question: str, mode: str = "bundle",
        toolbox: Toolbox | None = None,
        on_call: Callable[[int, ToolCall | None], None] | None = None) -> Run:
    folder = config.agent.bundles if mode == "bundle" else config.agent.raw_docs
    if not (folder / database).is_dir():
        raise ValueError(f"No {mode} docs for a database called {database!r} in {folder}")
    if toolbox is None:
        toolbox = Toolbox(folder / database if mode == "bundle" else None)
    run = Run(question, database, mode, config.model.name,
              started_at=datetime.now().astimezone().isoformat(timespec="seconds"))
    messages = [
        {"role": "system", "content": system_prompt(config, database, mode, toolbox)},
        {"role": "user", "content": question},
    ]

    try:
        for _ in range(config.agent.max_steps):
            reply = chat(config.model, messages, tools=toolbox.schemas())
            step = Step(reply)
            run.steps.append(step)
            messages.append(assistant_message(reply))

            if not reply.tool_calls:
                if on_call:
                    on_call(len(run.steps), None)
                messages.append({"role": "user", "content":
                                 "Keep using the tools. When you are done, call submit_answer."})
                continue

            for call in reply.tool_calls:
                result = toolbox.run(call["name"], call["arguments"])
                step.calls.append(ToolCall(call["name"], call["arguments"], result))
                if on_call:
                    on_call(len(run.steps), step.calls[-1])
                messages.append({"role": "tool", "tool_call_id": call["id"], "content": result})

            if toolbox.answer:
                run.answer = toolbox.answer
                run.stop_reason = "submitted" if toolbox.answer["sql"].strip() else "declined"
                return run
        run.stop_reason = "step_limit"
    except Exception as error:  # a failed API call, a malformed reply, ...
        run.stop_reason = "error"
        run.error = f"{type(error).__name__}: {error}"
    return run


def show_call(number: int, call: ToolCall | None) -> None:
    """Print one step as soon as it happens."""
    if call is None:
        detail, name = "(answered in text, nudged to use a tool)", ""
    elif call.name == "submit_answer":
        detail, name = "(final answer)", call.name
    else:
        name = call.name
        detail = call.arguments.get("path") or call.arguments.get("sql", "").split("\n")[0][:60]
        if call.result.startswith("Error"):
            detail += "  -> error"
    print(f"step {number}: {name:<14} {detail}", flush=True)


def load_task(config: Config, instance_id: str) -> dict:
    path = config.agent.raw_docs / "livesqlbench_data.jsonl"
    for line in path.read_text(encoding="utf-8").splitlines():
        task = json.loads(line)
        if task["instance_id"] == instance_id:
            return task
    raise SystemExit(f"No task called {instance_id} in {path}")


def question_from_args(config: Config, args: list[str]) -> tuple[str, str, str | None]:
    """A benchmark task ID (disaster_1), or a database and your own question."""
    if len(args) == 1:
        task = load_task(config, args[0])
        return task["selected_database"], task["query"], task["instance_id"]
    return args[0], args[1], None


if __name__ == "__main__":
    config = load_config()
    database, question, _ = question_from_args(
        config, [a for a in sys.argv[1:] if not a.startswith("--")])
    mode = "raw" if "--raw" in sys.argv else "bundle"
    run = ask(config, database, question, mode, on_call=show_call)

    print(f"\nstopped: {run.stop_reason} {run.error}".rstrip())
    if run.answer:
        print(f"\nanswer: {run.answer['answer']}\n\n{run.answer['sql']}".rstrip())

Notice seven things:

A step is one call to the model, not one tool call. A model can ask for several files in one turn, and max_steps (15 in config.yaml) counts turns. That's what costs time and tokens.
The instructions are the same in both modes. Only the knowledge part changes. In bundle mode, the agent gets read_file and is told how to navigate: start at index.md, read one folder index, open only what the question needs. The agent never gets a list of all 64 files. That's progressive disclosure, and these few lines are what ask for it. In raw mode, the three raw files are pasted into the prompt and read_file is gone. We use raw mode later to compare the two.
The agent is told what's out of scope. A greeting or a general request for help gets an answer straight away, with no files read and no SQL. So does a question about data the database doesn't hold.
If the model answers in plain text instead of calling a tool, we nudge it and the loop goes on. The step still counts.
Nothing crashes the run. A failed API call ends it with stop_reason set to error and the message saved. Every run ends as submitted, declined (an answer without SQL), step_limit or error. A database name with no bundle stops before the first model call.
You see each step as it happens. ask() takes an optional on_call function and calls it after every tool call. The command passes show_call, which prints one line. Without it, the terminal stays empty until the run ends.
Run keeps everything: each model reply with its token counts, and each tool call with its result. The next step turns that into a trace.

The block at the bottom takes either a benchmark task ID, such as disaster_1, or a database name and your own question. We'll ask the question from the opening in the next step, once every run leaves a record.

When the request isn't a data question

First, check that the agent knows when not to start. Ask it something that isn't a question about the data:

uv run -m okf_agent.agent disaster "can you help me with an sql generation task"

Output:

step 1: read_file      index.md
step 2: submit_answer  (final answer)

stopped: declined

answer: I can help you generate SQL queries for the disaster database. Please provide a specific question about the data, such as requesting certain information, calculations, or classifications defined in the knowledge base.

The agent read the root index once to see what the database covers, then declined in its second step and said what it can help with. The instructions say to decline without reading anything, so that one read is a small, visible cost. Without the out-of-scope rule, nothing tells the agent to stop, and it can spend all its steps reading files in search of something to answer.

For a real question, the agent would now hand in SQL that nobody has run, so it may not even be valid. Before we give the agent a database, we make every run leave a record we can read and compare.

Record every step

The steps on screen disappear when you close the terminal. To compare runs, such as the same question with and without the bundle, or one model against another, every run has to leave a record. We save each run as one line of JSON in runs/runs.jsonl.

Create okf_agent/trace.py:

"""Turn a finished run into a record: what it read, what it ran, what it cost."""

import json
import sys
from collections import Counter
from pathlib import Path

from okf_agent.agent import Run, ask, question_from_args, show_call
from okf_agent.config import load_config


def summarize(run: Run, **extra) -> dict:
    replies = [step.reply for step in run.steps]
    calls = [call for step in run.steps for call in step.calls]
    reads = [c for c in calls if c.name == "read_file"]
    queries = [c for c in calls if c.name == "run_sql"]
    costs = [r.cost_usd for r in replies if r.cost_usd is not None]

    return {
        **extra,
        "database": run.database,
        "mode": run.mode,
        "model": run.model,
        "question": run.question,
        "started_at": run.started_at,
        "stop_reason": run.stop_reason,
        "error": run.error,
        "llm_calls": len(replies),
        "tool_calls": dict(Counter(c.name for c in calls)),
        "files_opened": [c.arguments.get("path") for c in reads if not c.result.startswith("Error")],
        "failed_reads": sum(c.result.startswith("Error") for c in reads),
        "sql_attempts": len(queries),
        "sql_errors": sum(c.result.startswith("Error") for c in queries),
        "input_tokens": sum(r.input_tokens for r in replies),
        "output_tokens": sum(r.output_tokens for r in replies),
        "reasoning_tokens": sum(r.reasoning_tokens for r in replies),
        "cost_usd": sum(costs) if costs else None,
        "model_seconds": round(sum(r.seconds for r in replies), 2),
        "rate_limit_wait_seconds": sum(r.waited_seconds for r in replies),
        "answer": run.answer,
    }


def save(record: dict, folder: Path) -> Path:
    folder.mkdir(parents=True, exist_ok=True)
    path = folder / "runs.jsonl"
    with path.open("a", encoding="utf-8") as file:
        file.write(json.dumps(record, ensure_ascii=False) + "\n")
    return path


def report(record: dict) -> str:
    tools = ", ".join(f"{name} {count}" for name, count in record["tool_calls"].items())
    cost = "unknown" if record["cost_usd"] is None else f"${record['cost_usd']:.4f}"
    return "\n".join([
        f"stopped:       {record['stop_reason']} {record['error']}".rstrip(),
        f"model calls:   {record['llm_calls']}",
        f"tool calls:    {tools or 'none'}",
        f"files opened:  {len(record['files_opened'])} ({record['failed_reads']} failed reads)",
        f"sql attempts:  {record['sql_attempts']} ({record['sql_errors']} errors)",
        f"tokens:        {record['input_tokens']:,} in, {record['output_tokens']:,} out "
        f"({record['reasoning_tokens']:,} reasoning)",
        f"cost:          {cost}",
        f"model time:    {record['model_seconds']}s "
        f"(+{record['rate_limit_wait_seconds']}s waiting for rate limits)",
    ])


if __name__ == "__main__":
    config = load_config()
    database, question, task_id = question_from_args(
        config, [a for a in sys.argv[1:] if not a.startswith("--")])
    mode = "raw" if "--raw" in sys.argv else "bundle"
    run = ask(config, database, question, mode, on_call=show_call)

    record = summarize(run, task_id=task_id)
    path = save(record, config.agent.runs)
    print(f"\n{report(record)}\n\nrecord saved to {path}")

Notice five things:

Tokens are added up over every model call, and input tokens grow with each step. Every call sends the whole conversation again: the instructions, the tool descriptions and everything the agent has read so far. An extra step doesn't just cost its own reply; it resends all of that.
files_opened lists only the reads that worked. Failed reads, like a wrong guess at a file name, are counted separately, so a run's wrong turns stay visible.
sql_attempts and sql_errors are zero for now. The fields are already there, so every record has the same shape once the agent can run SQL.
Model time leaves out time spent waiting for rate limits. It still depends on the provider's load, so compare it only between runs on the same model and provider.
The record is plain JSON Lines. Each run is appended as one line, and any tool (Python, pandas, jq) can read the file later. If you also want a trace UI, the provider guide shows how to send every model call to Langfuse or LangSmith as well.

The block at the bottom runs a question with live steps, saves the record and prints a summary. Ask the question from the opening of this post, by its ID:

uv run -m okf_agent.trace disaster_1

Output:

step 1: read_file      index.md
step 2: read_file      knowledge/index.md
step 3: read_file      knowledge/calculation/resource-utilization-ratio.md  -> error
step 4: read_file      knowledge/calculation/index.md  -> error
step 5: read_file      knowledge/index.md
step 6: read_file      knowledge/calculation/resource-utilization-ratio.md  -> error
step 7: read_file      tables/index.md
step 8: read_file      knowledge/resource-utilization-ratio.md
step 9: read_file      knowledge/resource-utilization-classification.md
step 10: read_file      tables/distributionhubs.md
step 11: submit_answer  (final answer)

stopped:       submitted
model calls:   11
tool calls:    read_file 10, submit_answer 1
files opened:  7 (3 failed reads)
sql attempts:  0 (0 errors)
tokens:        39,280 in, 1,775 out (1,197 reasoning)
cost:          unknown
model time:    15.1s (+0.0s waiting for rate limits)

record saved to runs\runs.jsonl

This run took 11 model calls, and three of its reads failed. All three looked for a knowledge/calculation/ folder that doesn't exist: the agent took the # Calculation heading in the knowledge index for a folder, and went back to the same guess after the first error. The record shows exactly what that wrong turn cost.

Compare the input tokens with the reading cost we estimated for this question in the previous part: about 3,600 tokens for the files on the shortest path. This run sent 39,280 input tokens, about eleven times more. The files are only part of it. Every call resends the instructions, the tool descriptions and everything read so far, and each wrong turn adds one more call that resends it all again.

Every run now leaves a record, but the SQL in it has never been run. Next, we give the agent a database, safely.

Start the database with a read-only login for the agent

Until now, the agent's SQL was only a proposal. To run it, the agent needs to log in to the database. The benchmark's Docker setup has a single login, root, and it's a superuser: it can drop tables, read files on the database server and even run shell commands there. SQL written by a model must never run as that user.

So before the agent touches the database, we create a second user, agent_ro, that can only read.

You need Docker installed and running; see the tutorial guide for the setup. Then create docker-compose.yml in the project root:

services:
  postgresql:
    image: docker.io/shawnxxh/bird-interact-postgresql@sha256:1ae45d7aa5d64dd8eb82e4058f56b4b9625d5035b9b9dc0d2afa0295d9d3053c
    container_name: livesqlbench_postgresql
    environment:
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - "5432:5432"

volumes:
  pgdata:

The image is pinned by its digest, the exact version this post was tested with. The login comes from your .env file, so no password is written in a committed file. POSTGRES_USER must be root, because the image's loading script logs in under that name.

Create okf_agent/prepare.py:

"""Start the benchmark database, and give the agent a login that can only read."""

import subprocess
import time

import psycopg
from psycopg import sql

from okf_agent.config import Config, load_config

CHECKS = [
    ("read a table", "SELECT count(*) FROM distributionhubs"),
    ("drop a table", "DROP TABLE distributionhubs"),
    ("create a table", "CREATE TABLE notes (note text)"),
    ("read a server file", "SELECT pg_read_file('/etc/passwd')"),
]


def database_names(config: Config) -> list[str]:
    """One database per benchmark folder, named like the folder."""
    folder = config.agent.raw_docs
    return sorted(p.name for p in folder.iterdir() if (p / f"{p.name}_schema.txt").exists())


def connect(config: Config, dbname: str, admin: bool = False) -> psycopg.Connection:
    db = config.database
    user, password = ((db.admin_user, db.admin_password) if admin
                      else (db.agent_user, db.agent_password))
    return psycopg.connect(host=db.host, port=db.port, dbname=dbname, user=user,
                           password=password, connect_timeout=5)


def start_database(config: Config, names: list[str], wait_minutes: int = 30) -> None:
    try:
        subprocess.run(["docker", "compose", "up", "-d"], check=True)
    except (FileNotFoundError, subprocess.CalledProcessError):
        raise SystemExit("Couldn't start the database with Docker. Is Docker installed "
                         "and running? See TUTORIAL.md.")

    print(f"waiting until all {len(names)} databases are loaded...", flush=True)
    deadline = time.monotonic() + wait_minutes * 60
    while time.monotonic() < deadline:
        try:
            with connect(config, "postgres", admin=True) as conn:
                found = {row[0] for row in conn.execute("SELECT datname FROM pg_database")}
            if set(names) <= found:
                return
        except psycopg.OperationalError as error:
            if "password authentication failed" in str(error):
                raise SystemExit("The database rejected POSTGRES_PASSWORD from .env. "
                                 "See 'Change the database password' in TUTORIAL.md.")
        time.sleep(5)
    raise SystemExit("The databases didn't finish loading. Check: docker compose logs")


def create_agent_user(config: Config, names: list[str]) -> None:
    user = sql.Identifier(config.database.agent_user)
    with connect(config, "postgres", admin=True) as conn:
        conn.autocommit = True
        exists = conn.execute("SELECT 1 FROM pg_roles WHERE rolname = %s",
                              [config.database.agent_user]).fetchone()
        conn.execute(sql.SQL("{} ROLE {} LOGIN PASSWORD {}").format(
            sql.SQL("ALTER" if exists else "CREATE"), user,
            sql.Literal(config.database.agent_password)))

    for name in names:
        with connect(config, name, admin=True) as conn:
            conn.autocommit = True
            database = sql.Identifier(name)
            conn.execute("REVOKE CREATE ON SCHEMA public FROM PUBLIC")
            conn.execute(sql.SQL("REVOKE TEMPORARY ON DATABASE {} FROM PUBLIC").format(database))
            conn.execute(sql.SQL("GRANT CONNECT ON DATABASE {} TO {}").format(database, user))
            conn.execute(sql.SQL("GRANT USAGE ON SCHEMA public TO {}").format(user))
            conn.execute(sql.SQL("GRANT SELECT ON ALL TABLES IN SCHEMA public TO {}").format(user))


def check_agent_user(config: Config, database: str = "disaster") -> None:
    """Try a few things as the agent's user. Every attempt is rolled back."""
    for label, query in CHECKS:
        conn = connect(config, database)
        try:
            cursor = conn.execute(query)
            result = cursor.fetchone()[0] if cursor.description else "done"
            print(f"  allowed  {label}: {result}")
        except psycopg.Error as error:
            print(f"  blocked  {label}: {str(error).strip().splitlines()[0]}")
        finally:
            conn.rollback()
            conn.close()


if __name__ == "__main__":
    config = load_config()
    names = database_names(config)
    start_database(config, names)
    create_agent_user(config, names)
    print(f"user {config.database.agent_user} can now read {len(names)} databases\n")
    print("what the agent's user can do in disaster:")
    check_agent_user(config)

Notice four things:

start_database starts the container and waits until every database exists. On its first start, the image loads all 18 benchmark databases. For each one it keeps a template copy, such as disaster_template, and creates a plain database, disaster, which is the one the agent reads.
create_agent_user gives agent_ro three rights per database: to connect, to see the public schema, and to SELECT from its tables. Running it again only updates the password, so it's safe to repeat.
The two REVOKE lines close gaps that PostgreSQL 14 leaves open. By default, every user may create tables in the public schema and create temporary tables. We take both away, so reading is all that's left.
check_agent_user tries a few things as agent_ro, and rolls back every attempt. Even if a right were set up wrongly, nothing would change in the database.

Run it:

uv run -m okf_agent.prepare

Output:

[+] Running 1/1
 ✔ Container livesqlbench_postgresql  Running
waiting until all 18 databases are loaded...
user agent_ro can now read 18 databases

what the agent's user can do in disaster:
  allowed  read a table: 999
  blocked  drop a table: must be owner of table distributionhubs
  blocked  create a table: permission denied for schema public
  blocked  read a server file: permission denied for function pg_read_file

The agent's user can read the table, and that's all. Dropping it fails because agent_ro doesn't own it. Creating a table fails because it may not write to the schema. Reading a file on the server fails because only superusers may do that.

These refusals come from PostgreSQL itself, not from our code. Whatever SQL a model writes, the database checks it against this user's rights before running it. Next, we give the agent a tool to run its queries as this user.

Run SQL safely

The database already protects itself: the agent's user can only read. That's the first layer. Before the agent gets a tool to run queries, we add three more in our own code, so that a mistake in one layer is caught by another.

Create okf_agent/database.py:

"""Run one SQL query as the read-only user, with limits, and return text the model can read."""

import sys
from decimal import Decimal

import psycopg

from okf_agent.config import DatabaseConfig, load_config


def cell(value) -> str:
    if isinstance(value, (Decimal, float)):
        return f"{float(value):.6g}"  # short numbers keep the result small
    text = "NULL" if value is None else str(value)
    return text if len(text) <= 100 else text[:97] + "..."


def run_query(db: DatabaseConfig, database: str, query: str) -> str:
    try:
        with psycopg.connect(host=db.host, port=db.port, dbname=database, user=db.agent_user,
                             password=db.agent_password, connect_timeout=5) as conn:
            conn.read_only = True  # every transaction starts as READ ONLY
            conn.execute(f"SET statement_timeout = {db.statement_timeout_s * 1000}")
            cursor = conn.execute(query)
            if cursor.description is None:
                conn.rollback()
                return "The statement ran, but returned no rows."
            columns = [column.name for column in cursor.description]
            total = cursor.rowcount
            rows = cursor.fetchmany(db.max_rows)
            conn.rollback()  # nothing is ever kept
    except psycopg.errors.QueryCanceled:
        return f"Error: the query ran longer than {db.statement_timeout_s} seconds and was stopped."
    except psycopg.Error as error:
        hint = getattr(error.diag, "message_hint", None)
        return f"Error: {str(error).strip()}" + (f"\nHint: {hint}" if hint else "")

    lines = [" | ".join(columns)]
    lines += [" | ".join(cell(value) for value in row) for row in rows]
    shown = f"{len(rows)} of {total} rows" if total > len(rows) else f"{total} row{'s' * (total != 1)}"
    return "\n".join(lines) + f"\n({shown})"


if __name__ == "__main__":
    print(run_query(load_config().database, sys.argv[1], sys.argv[2]))

Notice four things:

Every query runs in a read-only transaction, and is then rolled back. If the agent's user ever had more rights than it should, a statement that changes data would still fail, and nothing it did would be kept.
A query that runs longer than 30 seconds is stopped. A model can easily write a query that joins every row to every other row. The limit comes from statement_timeout_s in config.yaml.
The model sees at most 20 rows, plus the total row count. That's enough to check that a query does what it should, without filling the model's context, and paying for it in tokens, with thousands of rows. Numbers are shortened to six significant digits for the same reason.
Errors come back as text, with PostgreSQL's position marker and hint. The model reads them and fixes its query, just as it does with a wrong file name.

Run a query as the agent's user:

uv run -m okf_agent.database disaster "SELECT hubregistry, (hubutilpct / 100.0) * (storecapm3 / (storeavailm3 + 1)) AS rur FROM distributionhubs ORDER BY rur DESC"

Output:

hubregistry | rur
HUB_XVHV | 460.207
HUB_J1JI | 457.992
HUB_V8QF | 349.307
HUB_XT44 | 200.055
HUB_NHLC | 187.676
...
(20 of 999 rows)

The query ran as agent_ro, and the model would see 20 rows plus the total: 999 hubs. (The output is shortened here.)

Now try to change data:

uv run -m okf_agent.database disaster "DELETE FROM distributionhubs"

Output:

Error: cannot execute DELETE in a read-only transaction

The read-only transaction stops the DELETE before the database even checks the user's rights.

Give the agent the tool

The agent gets a third tool, run_sql. Replace okf_agent/tools.py:

"""The tools the agent can call: what the model is told about them, and the code behind them."""

import sys
from collections.abc import Callable
from pathlib import Path

import yaml

READ_FILE = {
    "type": "function",
    "function": {
        "name": "read_file",
        "description": (
            "Read one file from the knowledge bundle and return its text. The path is relative "
            "to the bundle root, for example 'index.md' or 'tables/distributionhubs.md'."
        ),
        "parameters": {
            "type": "object",
            "properties": {"path": {"type": "string", "description": "Path of the file to read."}},
            "required": ["path"],
        },
    },
}

RUN_SQL = {
    "type": "function",
    "function": {
        "name": "run_sql",
        "description": (
            "Run one read-only PostgreSQL query and return the result: the column names, "
            "at most 20 rows, and the total number of rows."
        ),
        "parameters": {
            "type": "object",
            "properties": {"sql": {"type": "string", "description": "The query to run."}},
            "required": ["sql"],
        },
    },
}

SUBMIT_ANSWER = {
    "type": "function",
    "function": {
        "name": "submit_answer",
        "description": (
            "Submit your final answer. Call it once, at the end, with the final SQL query "
            "and a short answer in plain language. If the request can't be answered with a "
            "SQL query over this database, leave sql empty and explain why in answer."
        ),
        "parameters": {
            "type": "object",
            "properties": {
                "sql": {"type": "string", "description": "The final PostgreSQL query, or ''."},
                "answer": {"type": "string", "description": "A short answer in plain language."},
            },
            "required": ["sql", "answer"],
        },
    },
}


class Toolbox:
    """The tools for one run. Without a bundle there is no read_file; without a database, no run_sql."""

    def __init__(self, bundle: Path | None, run_sql: Callable[[str], str] | None = None):
        self.bundle = bundle.resolve() if bundle else None
        self.run_sql = run_sql
        self.answer: dict | None = None

    def schemas(self) -> list[dict]:
        tools = [READ_FILE] if self.bundle else []
        tools += [RUN_SQL] if self.run_sql else []
        return tools + [SUBMIT_ANSWER]

    def run(self, name: str, arguments: dict) -> str:
        handlers = {"submit_answer": self.submit_answer}
        if self.bundle:
            handlers["read_file"] = self.read_file
        if self.run_sql:
            handlers["run_sql"] = self.run_sql
        if name not in handlers:
            return f"Error: there is no tool called {name}."
        try:
            return handlers[name](**arguments)
        except TypeError as error:
            return f"Error: wrong arguments for {name}: {error}"

    def read_file(self, path: str) -> str:
        target = (self.bundle / path.lstrip("/")).resolve()
        if not target.is_relative_to(self.bundle):
            return f"Error: {path} is not a file in the bundle."
        if target.is_dir():
            target = target / "index.md"
        if target.suffix != ".md":
            return f"Error: {path} is not a file in the bundle."
        if target.is_file():
            return target.read_text(encoding="utf-8")
        if target.name == "index.md" and target.parent.is_dir():
            return self.list_folder(target.parent)  # OKF index files are optional
        return f"Error: {path} does not exist. Check the folder's index.md for the right name."

    def list_folder(self, folder: Path) -> str:
        """Build an index from the frontmatter, for a folder that has no index.md."""
        name = "the bundle root" if folder == self.bundle else folder.relative_to(self.bundle).as_posix()
        lines = [f"(No index.md here. Files in {name}:)"]
        for item in sorted(folder.iterdir()):
            if item.is_dir():
                lines.append(f"* [{item.name}/]({item.name}/index.md) - folder")
            elif item.suffix == ".md":
                text = item.read_text(encoding="utf-8")
                meta = yaml.safe_load(text.split("---", 2)[1]) if text.startswith("---") else {}
                title = (meta or {}).get("title", item.stem)
                lines.append(f"* [{title}]({item.name}) - {(meta or {}).get('description', '')}")
        return "\n".join(lines)

    def submit_answer(self, sql: str, answer: str) -> str:
        self.answer = {"sql": sql, "answer": answer}
        return "Answer received."


if __name__ == "__main__":
    from okf_agent.config import load_config

    database, path = sys.argv[1], sys.argv[2]
    toolbox = Toolbox(load_config().agent.bundles / database)
    print(toolbox.run("read_file", {"path": path}))

What changed compared with the previous version:

A new tool, run_sql. Its description tells the model what it gets back, including the 20-row limit, so the model doesn't mistake 20 rows for the whole result.
Toolbox takes a function that runs SQL. The toolbox doesn't know about databases or passwords. It only calls the function it's given, which keeps each module to one job.
Both modes get run_sql. With the bundle or with the raw docs, the agent can test its query. The only difference between the two remains how the agent gets its knowledge.

Then make three small changes in okf_agent/agent.py. Import the new module at the top:

from okf_agent.config import Config, load_config
from okf_agent.database import run_query

In ask(), give the toolbox a function that runs SQL on this question's database:

    if toolbox is None:
        toolbox = Toolbox(folder / database if mode == "bundle" else None,
                          run_sql=lambda sql: run_query(config.database, database, sql))

And in show_call(), print a query on one line:

        detail = call.arguments.get("path") or " ".join(call.arguments.get("sql", "").split())[:70]

Nothing else changes. The instructions already tell the agent to run its query before it submits, as soon as run_sql is among its tools.

Ask the question again, now with a database:

uv run -m okf_agent.trace disaster_1

Output:

step 1: read_file      index.md
step 2: read_file      knowledge/index.md
step 3: read_file      knowledge/resource-utilization-ratio.md
step 4: read_file      knowledge/resource-utilization-classification.md
step 5: read_file      tables/distributionhubs.md
step 6: run_sql        WITH rur_calc AS ( SELECT hubregistry, (hubutilpct / 100.0) * (storeca
step 7: submit_answer  (final answer)

stopped:       submitted
model calls:   7
tool calls:    read_file 5, run_sql 1, submit_answer 1
files opened:  5 (0 failed reads)
sql attempts:  1 (0 errors)
tokens:        18,809 in, 1,268 out (710 reasoning)
cost:          unknown
model time:    13.53s (+0.0s waiting for rate limits)

record saved to runs\runs.jsonl

This time the agent took the shortest path: the root index, the knowledge index, the two rules, then the table. It ran its query once, without an error, and handed it in. With no wrong turns, it used 18,809 input tokens, less than half of the earlier run.

Same question, same bundle, a different path. One run shows what can happen, not what usually happens. To know how often the agent finds the right knowledge, and how often its SQL is correct, it has to answer many questions, and its answers have to be checked against the benchmark's answer key.

Put it all behind one command

Every piece now works on its own: prepare starts the database, the agent answers, the trace records. But nobody wants to remember a list of commands. okf-agent works the way coding agents like Claude Code do: you type its name, it gets everything ready, and you ask your question.

For a readable terminal, we add one small package, rich. It draws the spinner while the model thinks, colors the SQL and frames the result. Add it to the dependencies in pyproject.toml and install it:

    "rich==15.0.0",

uv sync

Create okf_agent/cli.py:

"""The okf-agent command: get everything ready, then answer questions."""

import argparse

import litellm
from rich.console import Console
from rich.panel import Panel
from rich.syntax import Syntax

from okf_agent import prepare
from okf_agent.agent import ask, load_task, show_call
from okf_agent.config import Config, load_config
from okf_agent.database import run_query
from okf_agent.trace import save, summarize
from okf_compiler.cli import compile_database

console = Console()
HELP = "Type a question about the data. Commands: /db <name>, /raw, /bundle, /exit"


def get_ready(config: Config) -> None:
    """Everything the agent needs. Safe to run every time."""
    needed = litellm.validate_environment(model=config.model.name)["missing_keys"]
    missing = [key for key in needed if key.endswith("_KEY")]
    if missing:
        raise SystemExit(f"Add {', '.join(missing)} to your .env file (see PROVIDERS.md).")

    names = prepare.database_names(config)
    for name in names:
        if not (config.agent.bundles / name / "index.md").exists():
            with console.status(f"compiling the {name} bundle..."):
                compile_database(name)

    with console.status("starting the database..."):
        prepare.start_database(config, names)
        prepare.create_agent_user(config, names)


def answer(config: Config, database: str, question: str, mode: str,
           task_id: str | None = None) -> None:
    with console.status("thinking..."):
        run = ask(config, database, question, mode, on_call=show_call)
    record = summarize(run, task_id=task_id)
    path = save(record, config.agent.runs)

    if run.answer:
        console.print(f"\n[bold]answer:[/] {run.answer['answer']}")
        if run.answer["sql"].strip():
            console.print(Syntax(run.answer["sql"], "sql", word_wrap=True))
            result = run_query(config.database, database, run.answer["sql"])
            console.print(Panel(result, title="result", title_align="left"))
    if run.stop_reason not in ("submitted", "declined"):
        console.print(f"[red]stopped: {run.stop_reason} {run.error}[/]")

    calls, tokens = record["llm_calls"], record["input_tokens"] + record["output_tokens"]
    console.print(f"[dim]{calls} model call{'s' * (calls != 1)} · {tokens:,} tokens · "
                  f"{record['sql_errors']} SQL errors · saved to {path}[/]\n")


def chat(config: Config, database: str, mode: str) -> None:
    console.print(f"[bold]okf-agent[/] · {config.model.name}\n{HELP}\n")
    while True:
        try:
            line = console.input(f"[bold cyan]{database}[/] ({mode})> ").strip()
        except (EOFError, KeyboardInterrupt):
            break
        if line in ("/exit", "/quit"):
            break
        elif line.startswith("/db "):
            database = line.split(maxsplit=1)[1]
        elif line in ("/raw", "/bundle"):
            mode = line[1:]
        elif line.startswith("/") or not line:
            console.print(HELP)
        else:
            try:
                answer(config, database, line, mode)
            except ValueError as error:  # for example, a database with no bundle
                console.print(f"[red]{error}[/]")


def main() -> None:
    parser = argparse.ArgumentParser(prog="okf-agent", description="Ask a database a question.")
    parser.add_argument("--db", default="disaster", help="database to ask (default: disaster)")
    parser.add_argument("--raw", action="store_true", help="paste the raw docs instead of the bundle")
    commands = parser.add_subparsers(dest="command")
    one = commands.add_parser("ask", help="ask one question and exit")
    one.add_argument("question", nargs="?", help="your question")
    one.add_argument("--db", default=argparse.SUPPRESS, help="database to ask")
    one.add_argument("--raw", action="store_true", default=argparse.SUPPRESS,
                     help="paste the raw docs instead of the bundle")
    one.add_argument("--task", help="a benchmark task ID, such as disaster_1")
    args = parser.parse_args()
    if args.command == "ask" and not (args.task or args.question):
        parser.error("ask needs a question or --task")

    config = load_config()
    get_ready(config)
    mode = "raw" if args.raw else "bundle"

    if args.command != "ask":
        chat(config, args.db, mode)
    elif args.task:
        task = load_task(config, args.task)
        answer(config, task["selected_database"], task["query"], mode, task_id=args.task)
    else:
        answer(config, args.db, args.question, mode)


if __name__ == "__main__":
    main()

Notice four things:

get_ready runs every time, and is safe to repeat. It checks that the model's API key is set (LiteLLM knows which key each provider needs), compiles any missing bundle with the compiler from the previous part, starts the database and sets up the read-only user. On later runs, each step finds its work already done.
The steps print above a spinner. answer runs the agent inside console.status, and show_call, the same function as before, prints each step as it happens.
After the agent submits, its SQL runs once more, to show you the result. The record is saved before that, so this extra query isn't counted in the agent's metrics.
Every question is saved to runs/runs.jsonl, exactly as with the trace command. The prompt and ask are only two ways into the same ask() function.

You can use it in two ways. Start the prompt and type your questions:

uv run okf-agent

The prompt always shows two things: the database you're asking about and where the agent gets its knowledge, for example disaster (bundle)>. Anything you type is a question, unless it starts with /:

Command	What it does
`/db <name>`	Switches the database your next questions are about. The name is one of the 18 benchmark databases, the same as the folder names in `data/livesqlbench-base-lite`, for example `/db news` or `/db credit`.
`/raw`	The agent answers without the bundle: the database's three raw documentation files are pasted into its instructions, and it has no `read_file` tool. Use it to compare the two ways of giving an agent knowledge on the same question.
`/bundle`	Back to the default: the agent finds its knowledge in the OKF bundle with `read_file`.
`/exit`	Quits the prompt.

You can also ask one question and exit, which is handy in scripts. --db and --raw do the same as /db and /raw:

uv run okf-agent ask "Which hubs have the highest Resource Utilization Ratio?" --db disaster
uv run okf-agent ask --task disaster_1

See every option:

uv run okf-agent --help

Output:

usage: okf-agent [-h] [--db DB] [--raw] {ask} ...

Ask a database a question.

positional arguments:
  {ask}
    ask       ask one question and exit

options:
  -h, --help  show this help message and exit
  --db DB     database to ask (default: disaster)
  --raw       paste the raw docs instead of the bundle

Ask your first question

Everything is in place. Start the prompt:

uv run okf-agent

Then type the question from the opening of this post:

disaster (bundle)> I need to analyze all distribution hubs based on their Resource Utilization Ratio. Please show the hub registry ID, the calculated RUR value, and their Resource Utilization Classification. Sort the results by RUR from highest to lowest.

Output (the result table is shortened here):

step 1: read_file      index.md
step 2: read_file      knowledge/index.md
step 3: read_file      knowledge/calculation/resource-utilization-ratio.md  -> error
step 4: read_file      knowledge/calculation/index.md  -> error
step 5: read_file      knowledge/index.md
step 6: read_file      knowledge/resource-utilization-ratio.md
step 7: read_file      knowledge/resource-utilization-classification.md
step 8: read_file      tables/distributionhubs.md
step 9: run_sql        WITH rur_calc AS ( SELECT hubregistry, (hubutilpct/100.0) * (storecapm
step 10: submit_answer  (final answer)

answer: The query returns all distribution hubs with their Resource Utilization Ratio (RUR) and classification, sorted from highest to lowest RUR.
WITH rur_calc AS (
  SELECT
    hubregistry,
    (hubutilpct/100.0) * (storecapm3 / (storeavailm3 + 1.0)) AS rur
  FROM distributionhubs
)
SELECT
  hubregistry,
  rur,
  CASE
    WHEN rur > 5 THEN 'High Utilization'
    WHEN rur >= 2 AND rur <= 5 THEN 'Moderate Utilization'
    ELSE 'Low Utilization'
  END AS resource_utilization_classification
FROM rur_calc
ORDER BY rur DESC;
╭─ result ─────────────────────────────────────────────────╮
│ hubregistry | rur | resource_utilization_classification  │
│ HUB_XVHV | 460.207 | High Utilization                    │
│ HUB_J1JI | 457.992 | High Utilization                    │
│ HUB_V8QF | 349.307 | High Utilization                    │
│ ...                                                      │
│ (20 of 999 rows)                                         │
╰──────────────────────────────────────────────────────────╯
10 model calls · 37,273 tokens · 0 SQL errors · saved to runs\runs.jsonl

Read the answer from top to bottom:

The steps show which files the agent opened, and in what order. The knowledge this question needs is in two business rules and one table.
The SQL can be checked against those two rules: the formula (hubutilpct / 100) × (storecapm3 / (storeavailm3 + 1)), and the thresholds 5 and 2 for the three classes.
The result is real rows from the database. Check one by hand: a hub with an RUR above 5 must be labeled High Utilization.
The last line says what the answer cost and where its record is saved.

In this run, the SQL uses rule 10's formula exactly, and rule 50's thresholds: above 5 is High, 2 to 5 is Moderate, below 2 is Low. The top hub's RUR is 460.207, far above 5, so High Utilization is right.

The run also shows the same wrong turn we saw earlier: in steps 3 and 4, the agent looked for a knowledge/calculation/ folder that doesn't exist, then went back to the knowledge index and found the right files. Those two failed reads cost two extra model calls, and they show up in the 37,273 tokens on the last line.

This is the question the schema alone couldn't answer. No column is called "Resource Utilization Ratio", and nothing in the schema says where "High Utilization" starts. The agent looked both up in the bundle, wrote the query, and ran it before it answered.

But does the bundle make the difference, or would the model do just as well with the raw documentation? That's the next question.

Ask the same question without the bundle

The usual alternative to a knowledge layer is to paste all the documentation into the prompt. Our agent can do exactly that. In raw mode, everything stays the same: the instructions, the model, the database and the run_sql tool. Only one thing changes: instead of reading the bundle with read_file, the agent gets the database's three raw documentation files pasted into its instructions.

That makes raw mode a fair comparison. If anything, it gets slightly more information than the bundle: the raw schema file also contains three sample rows per table, which the bundle leaves out.

In the same prompt, switch to raw mode and ask the same question:

disaster (bundle)> /raw
disaster (raw)> I need to analyze all distribution hubs based on their Resource Utilization Ratio. Please show the hub registry ID, the calculated RUR value, and their Resource Utilization Classification. Sort the results by RUR from highest to lowest.

Output (the result table is shortened here):

step 1: run_sql        SELECT hubregistry, (hubutilpct/100.0) * (storecapm3 / (storeavailm3 +
step 2: submit_answer  (final answer)

answer: The query returns each distribution hub's ID, Resource Utilization Ratio (RUR), and classification (High, Moderate, or Low Utilization) sorted by RUR from highest to lowest.
SELECT
    hubregistry,
    (hubutilpct/100.0) * (storecapm3 / (storeavailm3 + 1)) AS rur,
    CASE
        WHEN (hubutilpct/100.0) * (storecapm3 / (storeavailm3 + 1)) > 5 THEN 'High Utilization'
        WHEN (hubutilpct/100.0) * (storecapm3 / (storeavailm3 + 1)) >= 2 AND (hubutilpct/100.0) * (storecapm3 / (storeavailm3 + 1)) <= 5 THEN 'Moderate Utilization'
        ELSE 'Low Utilization'
    END AS resource_utilization_classification
FROM distributionhubs
ORDER BY rur DESC;
╭─ result ─────────────────────────────────────────────────╮
│ hubregistry | rur | resource_utilization_classification  │
│ HUB_XVHV | 460.207 | High Utilization                    │
│ HUB_J1JI | 457.992 | High Utilization                    │
│ HUB_V8QF | 349.307 | High Utilization                    │
│ ...                                                      │
│ (20 of 999 rows)                                         │
╰──────────────────────────────────────────────────────────╯
2 model calls · 36,973 tokens · 0 SQL errors · saved to runs\runs.jsonl

Same formula, same thresholds, same rows. With all the documentation already in its instructions, the agent didn't need to look anything up: it wrote the query, ran it once, and handed it in.

Here is how the runs compare. The middle column is the bundle run from the step where the agent first ran SQL, when it took the shortest path:

	Bundle (previous step)	Bundle, shortest path	Raw docs
Model calls	10	7	2
Failed reads	2	0	–
Total tokens	37,273	20,077	36,973
SQL errors	0	0	0

For this question, the bundle didn't win. Raw mode found the same answer with about the same tokens as our latest bundle run, in a fifth of the model calls. The bundle saved tokens only when the agent took the shortest path, and then it used about half.

The two modes spend their tokens differently. In raw mode, the full documentation, about 15,800 tokens for disaster by the estimate from the previous part, is sent with every model call, whatever the question needs. Two calls make it cheap here. With the bundle, the agent pays only for the files it opens, but every extra step resends everything read so far, so wrong turns are expensive.

Which way wins depends on the size of the documentation and on how well the agent navigates, and one question can't settle that. disaster is a small database: its docs fit easily in a prompt. A company's documentation often doesn't, and then pasting everything stops being an option at all. To compare the two fairly, we need many questions, across databases of different sizes, checked against the answer key.

What's next

You now have a SQL agent you start with one command. It looks up business knowledge in an OKF bundle before it writes a query, runs that query as a read-only user, and records what every answer cost. Along the way, we saw three things:

The knowledge layer works, but navigation costs. In every run, the agent found the two rules and the table the question needs. Its wrong turns, though, cost full model calls, and the index layout invited the same wrong guess more than once.
Guard the database, not the SQL text. A read-only user, a read-only transaction, a timeout and a row limit stopped every write we tried, without inspecting a single query.
One run is an anecdote. The same question took 2 to 11 model calls depending on the run and the mode, and on this small database, pasting the raw docs was just as cheap.

In the next part, we put numbers on it. We run many benchmark questions through the same ask() function, with the bundle and with the raw docs, check every answer against the benchmark's answer key, and look closely at where each approach fails.

The code for this series is in the okf-sql-knowledge repository.

Sources

Open Knowledge Format (OKF) specification, v0.2 — Google Cloud. Concept files, frontmatter, index files (optional in a bundle).
Your Agents' Knowledge Needs a File Format. Meet OKF — the introduction to OKF and its frontmatter fields.
LiveSQLBench and its GitHub repository — BIRD team at HKU and Google Cloud. The evaluation/ folder documents the PostgreSQL Docker image: it loads the databases on first start, logs in as root, and keeps a template and a plain copy of each database.
LiveSQLBench-Base-Lite dataset — the 18 databases and the disaster_1 question used in this post (CC BY-SA 4.0).
LiteLLM documentation — one interface to many model providers, token usage and cost; NVIDIA NIM provider.
NVIDIA API catalog — the free hosted open models used in this post.
PostgreSQL 15 release notes — PostgreSQL 15 removed PUBLIC's right to create objects in the public schema; the benchmark runs PostgreSQL 14, where it still exists.
PostgreSQL 14: privileges and client settings, including statement_timeout.
psycopg 3 — the PostgreSQL driver (version 3.3.6).
Docker Compose: variable interpolation — how Compose reads .env.
uv, python-dotenv and rich — project tooling, secrets from .env, and the terminal interface.
okf-sql-knowledge — the code and guides for this series.

Turn Database Docs into Agent-Ready Knowledge (OKF Series, Part 3)

Lakhan Malviya — Sat, 03 Oct 2026 04:41:06 +0000

An AI agent that writes SQL for your database needs more than the schema. It also needs to know what your business terms mean: which columns make up a "Resource Utilization Ratio", or which three conditions turn an operation "high-risk". Most teams already have that knowledge written down somewhere, in a schema dump, a column dictionary or a page of metric definitions. The trouble is that it's not in a form an agent can find its way through.

Over this part and the next, we build the pieces that close that gap for one real database, and run them end to end:

A compiler. A small Python package that reads a database's raw documentation and writes it as an OKF bundle: one concept file per table and per business rule, linked to each other, with an index file in each folder.
An agent. A simple SQL agent on an open model. It reads the bundle the way you'd browse a wiki: the index first, then only the files the question needs. Then it writes the SQL and runs it.
A side-by-side run. We ask the same questions twice: once with the bundle, and once with all the raw documentation pasted into the prompt. Then we compare what the agent read, how many tokens it used, and the SQL it wrote.

This part builds the first piece, the compiler, and runs it on all 18 databases in the benchmark. The next part builds the agent and runs the comparison. By the end of both, you'll have a working system on your laptop: one database, its bundle, and an agent you can ask your own questions.

Our database is disaster, from LiveSQLBench, a public text-to-SQL benchmark: 10 tables and 54 business rules about disaster-response operations. If you followed the previous part, you wrote three of its concept files by hand. Now we generate all of them. If you didn't, everything you need is in this post.

Set up the project and the database

We need two things before writing any code: a place for the compiler to live, and a real database for the agent to run its SQL against.

The project layout

We add a Python package, okf_compiler, next to the data/ and bundles/ folders. (If you're starting here, set up the project and download the data as described in the tutorial guide.)

Here's where we're heading. Each module is written, in full, in the step that needs it:

okf-sql-knowledge/
├── data/
│   └── livesqlbench-base-lite/
├── bundles/
├── okf_compiler/
│   ├── __init__.py
│   ├── readers.py      read the three raw files into Python
│   ├── concept.py      write one concept file: frontmatter + body
│   ├── tables.py       build one concept file per table
│   ├── rules.py        build one concept file per business rule, with links
│   ├── indexes.py      write the index files and the log
│   └── cli.py          one command to compile a database
├── docker-compose.yml
├── requirements.txt
└── README.md`

Create the okf_compiler/ folder with an empty __init__.py inside it.

The database

The agent we build later runs real SQL, so it needs the real disaster database. The LiveSQLBench team publishes a PostgreSQL image with all 18 Base-Lite databases already loaded. You need Docker Desktop to run it; see the tutorial guide.

Create docker-compose.yml in the project folder:

services:
  postgres:
    image: docker.io/shawnxxh/bird-interact-postgresql:latest
    container_name: livesqlbench_postgresql
    environment:
      POSTGRES_USER: root
      POSTGRES_PASSWORD: "123123"
    ports:
      - "5432:5432"
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

Start it, and follow the log:

docker compose up -d
docker logs -f livesqlbench_postgresql

Wait until the log stops printing import messages, then press Ctrl+C. The container keeps running. You don't need the database until we build the agent, so you can move on to the next step while it loads.

Each benchmark database has _template added to its name, so ours is called disaster_template. Check that its tables are there:

docker exec -it livesqlbench_postgresql psql -U root -d disaster_template -c "\dt"

You should see the ten tables:

               List of relations
 Schema |            Name             | Type  | Owner
--------+-----------------------------+-------+-------
 public | beneficiariesandassessments | table | root
 public | coordinationandevaluation   | table | root
 public | disasterevents              | table | root
 public | distributionhubs            | table | root
 public | environmentandhealth        | table | root
 public | financials                  | table | root
 public | humanresources              | table | root
 public | operations                  | table | root
 public | supplies                    | table | root
 public | transportation              | table | root

The ten tables match the ten CREATE TABLE statements in disaster_schema.txt. The database holds the data, but nothing in it says how to turn hubutilpct into a Resource Utilization Ratio. That's what the bundle is for.

Port already in use? If PostgreSQL already runs on your laptop, change the port line to "5433:5432" and use port 5433 wherever this post connects to the database.

Read the raw files into Python

The compiler's first job is to turn three different file formats (a SQL text file, a JSON object and a JSON Lines file) into one set of Python objects. Every later step works with those objects and never touches the raw files again.

Create okf_compiler/readers.py:

"""Read the three raw LiveSQLBench files for one database into Python objects."""

import json
import re
from dataclasses import dataclass, field
from pathlib import Path


@dataclass
class Column:
    name: str
    sql_type: str
    nullable: bool
    meaning: str = ""
    json_fields: dict = field(default_factory=dict)  # only for JSONB columns


@dataclass
class ForeignKey:
    column: str
    ref_table: str
    ref_column: str


@dataclass
class Table:
    name: str
    columns: list[Column]
    primary_key: list[str]
    foreign_keys: list[ForeignKey]


@dataclass
class Rule:
    id: int
    name: str
    description: str
    definition: str
    kind: str
    depends_on: list[int]


COLUMN_LINE = re.compile(r"^(\w+)\s+(.+?)\s+(NOT NULL|NULL),?$")
PRIMARY_KEY = re.compile(r"PRIMARY KEY \(([^)]+)\)")
FOREIGN_KEY = re.compile(r"FOREIGN KEY \((\w+)\) REFERENCES (\w+)\((\w+)\)")


def read_schema(path: Path) -> list[Table]:
    tables = []
    for block in path.read_text(encoding="utf-8").split("CREATE TABLE ")[1:]:
        name = block.split('"')[1]
        body = block.split(");")[0]
        columns, primary_key, foreign_keys = [], [], []
        for line in body.splitlines()[1:]:
            line = line.strip()
            if m := COLUMN_LINE.match(line):
                columns.append(Column(m[1], m[2], m[3] == "NULL"))
            elif m := PRIMARY_KEY.search(line):
                primary_key = [c.strip() for c in m[1].split(",")]
            elif m := FOREIGN_KEY.search(line):
                foreign_keys.append(ForeignKey(m[1], m[2], m[3]))
        tables.append(Table(name, columns, primary_key, foreign_keys))
    return tables


def add_column_meanings(tables: list[Table], path: Path) -> list[str]:
    """Attach each column's meaning. Returns the columns that have none."""
    meanings = {}
    for key, value in json.loads(path.read_text(encoding="utf-8")).items():
        _, table, column = key.lower().split("|")
        meanings[(table, column)] = value

    missing = []
    for table in tables:
        for column in table.columns:
            value = meanings.get((table.name.lower(), column.name.lower()))
            if value is None:
                missing.append(f"{table.name}.{column.name}")
            elif isinstance(value, dict):  # JSONB column
                column.meaning = value.get("column_meaning", "")
                column.json_fields = value.get("fields_meaning", {})
            else:
                column.meaning = value
    return missing


def read_rules(path: Path) -> list[Rule]:
    rules = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        raw = json.loads(line)
        children = raw["children_knowledge"]
        rules.append(Rule(
            id=raw["id"],
            name=raw["knowledge"],
            description=raw["description"],
            definition=raw["definition"],
            kind=raw["type"],
            depends_on=children if isinstance(children, list) else [],
        ))
    return rules

The four dataclasses are the shape every later module relies on: a Table with its Columns and ForeignKeys, and a Rule.

Notice four things the raw data forced on us:

The schema is split on CREATE TABLE, not on blank lines. Each table is followed by sample rows, and a table with no sample rows would otherwise run into the next one.
Names are matched in lowercase. The files mix cases: the column impactmetrics in the schema is impactMetrics in the descriptions, and some schemas have capitalized column names.
JSONB columns have a description for every nested field, not just one sentence. We keep those field descriptions in json_fields, because an agent can't query a JSON field it doesn't know exists.
children_knowledge is either a list or -1. We turn it into depends_on, which is always a list, so later code never has to check for -1.

add_column_meanings also returns the columns it couldn't find a description for, instead of failing quietly. It's our first report of what the source documentation is missing.

Check what was read

Create inspect_db.py in the project root, next to docker-compose.yml (not inside okf_compiler/). It reads one database and prints a summary:

import sys
from pathlib import Path

from okf_compiler.readers import add_column_meanings, read_rules, read_schema

name = sys.argv[1]
folder = Path("data/livesqlbench-base-lite") / name

tables = read_schema(folder / f"{name}_schema.txt")
missing = add_column_meanings(tables, folder / f"{name}_column_meaning_base.json")
rules = read_rules(folder / f"{name}_kb.jsonl")

print(f"tables:  {len(tables)}")
print(f"columns: {sum(len(t.columns) for t in tables)}")
print(f"columns without a meaning: {', '.join(missing) or 'none'}")
print(f"rules:   {len(rules)}")
print()
for table in tables:
    joins = ", ".join(fk.ref_table for fk in table.foreign_keys) or "-"
    print(f"{table.name:<28} {len(table.columns):>3} columns   joins: {joins}")

Run it for disaster:

python inspect_db.py disaster

The output:

tables:  10
columns: 123
columns without a meaning: none
rules:   54

beneficiariesandassessments   10 columns   joins: disasterevents, operations
disasterevents                 9 columns   joins: -
distributionhubs              11 columns   joins: disasterevents
environmentandhealth          13 columns   joins: disasterevents
operations                    12 columns   joins: disasterevents, distributionhubs
coordinationandevaluation     29 columns   joins: disasterevents, operations
humanresources                 4 columns   joins: disasterevents, operations
supplies                       4 columns   joins: disasterevents, distributionhubs
transportation                18 columns   joins: disasterevents, distributionhubs, supplies
financials                    13 columns   joins: disasterevents, operations

Ten tables, the same as the ten CREATE TABLE statements, and 54 rules, one per line of disaster_kb.jsonl. Every column has a description.

That isn't true for every database. Run it for mental:

python inspect_db.py mental

The output (the table list is cut):

tables:  9
columns: 105
columns without a meaning: patients.clinleadref
rules:   63
...

One column, patients.clinleadref, has no description in the source. The compiler can't invent one, so that column will reach the bundle with its name and type only. It's the first gap we've found in the source documentation, and the compiler should report gaps like this rather than hide them.

Your project now looks like this:

okf-sql-knowledge/
├── data/
│   └── livesqlbench-base-lite/
├── bundles/
├── okf_compiler/
│   ├── __init__.py
│   └── readers.py
├── docker-compose.yml
├── inspect_db.py
├── requirements.txt
└── README.md

Generate the table concepts

Each table becomes one concept file: the file an agent opens when it needs a table's columns, their meanings and its joins.

A helper for every concept file

Every concept file the compiler writes has the same shape: YAML frontmatter, then a markdown body. One small module writes that shape, so the table and rule modules don't repeat it.

Create okf_compiler/concept.py:

"""Write one concept file: YAML frontmatter, then a markdown body."""

from datetime import datetime
from pathlib import Path

import yaml

COMPILER = "okf_compiler/0.1"
SOURCE_BASE = "https://huggingface.co/datasets/birdsql/livesqlbench-base-lite/blob/main"


def generated() -> dict:
    now = datetime.now().astimezone().isoformat(timespec="seconds")
    return {"by": COMPILER, "at": now}


def source(database: str, filename: str, title: str) -> dict:
    return {
        "id": filename.split(".")[0].removeprefix(f"{database}_"),
        "resource": f"{SOURCE_BASE}/{database}/{filename}",
        "title": title,
    }


def write_concept(path: Path, frontmatter: dict, body: str) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    header = yaml.safe_dump(frontmatter, sort_keys=False, allow_unicode=True, width=1000)
    path.write_text(f"---\n{header}---\n\n{body.strip()}\n", encoding="utf-8")

Notice two things about the frontmatter it produces. (What each field means is covered in the introduction to OKF.)

generated.by is the compiler and its version, not a person. A reader of the bundle can tell these files were written by okf_compiler/0.1, and when.
There is no verified field. Nobody has reviewed these files, and the bundle says so. If someone checks a concept later, they add verified to that file.

The table module

Create okf_compiler/tables.py:

"""Build one concept file per table."""

from pathlib import Path

from okf_compiler.concept import generated, source, write_concept
from okf_compiler.readers import Table


def describe(table: Table) -> str:
    keys = set(table.primary_key) | {fk.column for fk in table.foreign_keys}
    names = [c.name for c in table.columns if c.name not in keys]
    text = f"{len(table.columns)} columns: {', '.join(names)}."
    joins = sorted({fk.ref_table for fk in table.foreign_keys})
    if joins:
        text += f" Joins to {', '.join(joins)}."
    return text


def cell(text: str) -> str:
    return text.replace("|", "\\|").replace("\n", " ")


def json_field_lines(prefix: str, fields: dict) -> list[str]:
    lines = []
    for name, value in fields.items():
        path = f"{prefix}.{name}"
        if isinstance(value, dict):
            lines += json_field_lines(path, value)
        else:
            lines.append(f"* `{path}`: {value}")
    return lines


def table_body(table: Table, related: list[tuple[str, str]]) -> str:
    parts = ["# Schema", "", "| Column | Type | Meaning |", "|---|---|---|"]
    for c in table.columns:
        sql_type = c.sql_type + (", primary key" if c.name in table.primary_key else "")
        parts.append(f"| `{c.name}` | {sql_type} | {cell(c.meaning)} |")

    json_lines = []
    for c in table.columns:
        json_lines += json_field_lines(c.name, c.json_fields)
    if json_lines:
        parts += ["", "# JSON fields", "", *json_lines]

    if table.foreign_keys:
        parts += ["", "# Joins", ""]
        for fk in table.foreign_keys:
            parts.append(
                f"* `{fk.column}` references `{fk.ref_column}` in "
                f"[{fk.ref_table}](/tables/{fk.ref_table}.md)."
            )

    if related:
        parts += ["", "# Related knowledge", ""]
        for title, slug in related:
            parts.append(f"* [{title}](/knowledge/{slug}.md)")
    return "\n".join(parts)


def write_table(table: Table, database: str, bundle: Path, related: list[tuple[str, str]]) -> None:
    frontmatter = {
        "type": "PostgreSQL Table",
        "title": table.name,
        "description": describe(table),
        "tags": [database],
        "generated": generated(),
        "sources": [
            source(database, f"{database}_schema.txt", f"{database} schema (LiveSQLBench)"),
            source(database, f"{database}_column_meaning_base.json",
                   f"{database} column descriptions (LiveSQLBench)"),
        ],
    }
    write_concept(bundle / "tables" / f"{table.name}.md", frontmatter, table_body(table, related))

The body keeps every column's meaning word for word, because that's where details like enum values live (warehousestate is Fair, Excellent, Good or Poor). JSONB columns get an extra section that lists every nested field as a path, such as impactmetrics.population.affected, ready to use in a query.

Design choice: table descriptions come from column names, not an LLM.
description is the one line an agent sees in the index before deciding whether to open the file. A person would write "one row per distribution hub, with its capacity and storage". A compiler can't write that sentence without an LLM, and an LLM would add wording that isn't in the source docs, which would give the bundle knowledge the other setups don't get. So describe lists the table's column names and the tables it joins to: plain, but taken straight from the source. If the agent struggles to pick the right tables, this is the first thing to improve.

write_table also takes a list of related business rules. It's empty for now; the next step fills it, so each table links to the rules that use its columns.

The command

Create okf_compiler/cli.py. For now it only writes the table concepts; we extend it in the next steps:

"""Compile one LiveSQLBench database into an OKF bundle."""

import shutil
import sys
from pathlib import Path

from okf_compiler.readers import add_column_meanings, read_rules, read_schema
from okf_compiler.tables import write_table

DATA = Path("data/livesqlbench-base-lite")
OUT = Path("bundles/compiled")


def compile_database(name: str) -> None:
    folder = DATA / name
    tables = read_schema(folder / f"{name}_schema.txt")
    missing = add_column_meanings(tables, folder / f"{name}_column_meaning_base.json")
    rules = read_rules(folder / f"{name}_kb.jsonl")

    bundle = OUT / name
    shutil.rmtree(bundle, ignore_errors=True)
    for table in tables:
        write_table(table, name, bundle, related=[])

    print(f"{name}: {len(tables)} table concepts written to {bundle}")
    if missing:
        print(f"  columns without a description: {', '.join(missing)}")


if __name__ == "__main__":
    compile_database(sys.argv[1])

The compiler writes to bundles/compiled/<database>, so it never overwrites a bundle you wrote by hand, and it clears that folder first, so the bundle always matches the current source files.

Run it:

python -m okf_compiler.cli disaster

The output:

disaster: 10 table concepts written to bundles\compiled\disaster

Open bundles/compiled/disaster/tables/distributionhubs.md. It should look like this (most columns cut):

---
type: PostgreSQL Table
title: distributionhubs
description: '11 columns: hubcaptons, hubutilpct, storecapm3, storeavailm3, coldstorecapm3, coldstoretempc, warehousestate, invaccpct, stockturnrate. Joins to disasterevents.'
tags:
- disaster
generated:
  by: okf_compiler/0.1
  at: '2026-10-03T00:03:25+05:30'
sources:
- id: schema
  resource: https://huggingface.co/datasets/birdsql/livesqlbench-base-lite/blob/main/disaster/disaster_schema.txt
  title: disaster schema (LiveSQLBench)
- id: column_meaning_base
  resource: https://huggingface.co/datasets/birdsql/livesqlbench-base-lite/blob/main/disaster/disaster_column_meaning_base.json
  title: disaster column descriptions (LiveSQLBench)
---

# Schema

| Column | Type | Meaning |
|---|---|---|
| `hubregistry` | character varying, primary key | A VARCHAR(20) primary key identifying each distribution hub record (e.g., 'HUB0001'). |
| `disteventref` | character varying | A VARCHAR(20) referencing DisasterEvents(DistRegistry), linking this hub to a specific disaster event (e.g., 'DIST0001'). |
... 9 more columns

# Joins

* `disteventref` references `distregistry` in [disasterevents](/tables/disasterevents.md).

Your project now looks like this:

okf-sql-knowledge/
├── data/
│   └── livesqlbench-base-lite/
├── bundles/
│   └── compiled/
│       └── disaster/
│           └── tables/          10 concept files
├── okf_compiler/
│   ├── __init__.py
│   ├── cli.py
│   ├── concept.py
│   ├── readers.py
│   └── tables.py
├── docker-compose.yml
├── inspect_db.py
├── requirements.txt
└── README.md

Generate the rule concepts and their links

The business rules are what the schema can't tell an agent: what "Resource Utilization Ratio" means, or which conditions make an operation "high-risk". Each rule becomes one concept file, linked to the rules it depends on and to the tables whose columns it uses.

Create okf_compiler/rules.py:

"""Build one concept file per business rule, and link rules to tables."""

import re
from pathlib import Path

from okf_compiler.concept import generated, source, write_concept
from okf_compiler.readers import Rule, Table

TYPE_NAMES = {
    "calculation_knowledge": "Calculation",
    "domain_knowledge": "Business Rule",
    "value_illustration": "Value Illustration",
}
WORD = re.compile(r"[A-Za-z_][A-Za-z0-9_]*")


def make_slugs(rules: list[Rule]) -> dict[int, str]:
    """A file name for every rule, e.g. 'resource-utilization-ratio'."""
    slugs, used = {}, set()
    for rule in rules:
        name = re.sub(r"\([^)]*\)", "", rule.name)
        slug = re.sub(r"[^a-z0-9]+", "-", name.lower()).strip("-")
        if slug in used:
            slug = f"{slug}-{rule.id}"
        used.add(slug)
        slugs[rule.id] = slug
    return slugs


def find_tables(rule: Rule, tables: list[Table]) -> dict[str, list[str]]:
    """Tables whose column names appear in the rule's name or definition."""
    words = {w.lower() for w in WORD.findall(f"{rule.name} {rule.definition}")}
    found = {}
    for table in tables:
        columns = [c.name for c in table.columns if c.name.lower() in words]
        if columns:
            found[table.name] = columns
    return found


def rule_body(rule: Rule, rules_by_id: dict[int, Rule], used_by: list[int],
              slugs: dict[int, str], tables: dict[str, list[str]]) -> str:
    parts = ["# Definition", "", rule.definition]

    if tables:
        parts += ["", "# Columns used", ""]
        for table, columns in tables.items():
            names = ", ".join(f"`{c}`" for c in columns)
            parts.append(f"* [{table}](/tables/{table}.md): {names}")

    for heading, ids in (("Depends on", rule.depends_on), ("Used by", used_by)):
        links = [f"* [{rules_by_id[i].name}](/knowledge/{slugs[i]}.md)"
                 for i in ids if i in rules_by_id]
        if links:
            parts += ["", f"# {heading}", "", *links]
    return "\n".join(parts)


def write_rules(rules: list[Rule], tables: list[Table], database: str,
                bundle: Path) -> dict[str, list[tuple[str, str]]]:
    """Write every rule concept. Returns, per table, the rules that use it."""
    slugs = make_slugs(rules)
    rules_by_id = {r.id: r for r in rules}
    used_by = {r.id: [] for r in rules}
    for rule in rules:
        for parent in rule.depends_on:
            if parent in used_by:
                used_by[parent].append(rule.id)

    related = {t.name: [] for t in tables}
    for rule in rules:
        found = find_tables(rule, tables)
        for table in found:
            related[table].append((rule.name, slugs[rule.id]))

        frontmatter = {
            "type": TYPE_NAMES.get(rule.kind, rule.kind),
            "title": rule.name,
            "description": rule.description,
            "tags": [database],
            "generated": generated(),
            "sources": [source(database, f"{database}_kb.jsonl",
                               f"{database} business rules (LiveSQLBench), rule {rule.id}")],
        }
        body = rule_body(rule, rules_by_id, used_by[rule.id], slugs, found)
        write_concept(bundle / "knowledge" / f"{slugs[rule.id]}.md", frontmatter, body)
    return related

Notice three things:

Each rule type gets a readable OKF type. calculation_knowledge becomes Calculation, domain_knowledge becomes Business Rule, and value_illustration becomes Value Illustration.
Dependencies become links in both directions. depends_on comes straight from the source. Used by is the reverse, built by the compiler, so an agent that lands on either rule can reach the other.
The definition is copied as it is, LaTeX included. Converting formulas to plain text would risk changing a formula, and LLMs read LaTeX well.

find_tables is the only guess the compiler makes. The source never says which table a rule belongs to. So we look for column names inside the rule's name and definition: rule 10 mentions hubutilpct, storecapm3 and storeavailm3, and all three are columns of distributionhubs. write_rules returns those matches per table, so each table concept can list the rules that use it.

One more detail: two rules can have the same name once the brackets are removed. make_slugs adds the rule's id to the second one, so no file overwrites another.

Update the command

Replace okf_compiler/cli.py:

"""Compile one LiveSQLBench database into an OKF bundle."""

import shutil
import sys
from pathlib import Path

from okf_compiler.readers import add_column_meanings, read_rules, read_schema
from okf_compiler.rules import find_tables, write_rules
from okf_compiler.tables import write_table

DATA = Path("data/livesqlbench-base-lite")
OUT = Path("bundles/compiled")


def compile_database(name: str) -> None:
    folder = DATA / name
    tables = read_schema(folder / f"{name}_schema.txt")
    missing = add_column_meanings(tables, folder / f"{name}_column_meaning_base.json")
    rules = read_rules(folder / f"{name}_kb.jsonl")

    bundle = OUT / name
    shutil.rmtree(bundle, ignore_errors=True)
    related = write_rules(rules, tables, name, bundle)
    for table in tables:
        write_table(table, name, bundle, related[table.name])

    unlinked = [r for r in rules if not find_tables(r, tables)]
    print(f"{name}: {len(tables)} tables, {len(rules)} rules written to {bundle}")
    if missing:
        print(f"  columns without a description: {', '.join(missing)}")
    if unlinked:
        print(f"  rules not linked to any table ({len(unlinked)}):")
        for rule in unlinked:
            print(f"    {rule.id}: {rule.name}")


if __name__ == "__main__":
    compile_database(sys.argv[1])

It now writes the rules first, because the table concepts need to know which rules link to them. It also prints a report of every rule it couldn't link to a table.

Run it:

python -m okf_compiler.cli disaster

The output:

disaster: 10 tables, 54 rules written to bundles\compiled\disaster
  rules not linked to any table (4):
    42: Sustainable Operation Excellence
    50: Resource Utilization Classification
    51: Environmental Impact Classification
    52: Community Resilience Classification

Four of the 54 rules are not linked to a table. Look at their names: three are classifications, such as rule 50, Resource Utilization Classification. Rule 50 never mentions a column. It says "RUR > 5", and RUR is rule 10. So it isn't lost: an agent reaches its table in two hops, through the Depends on link to rule 10, then rule 10's link to distributionhubs.

That's why the compiler reports these rules instead of guessing. A missing link costs the agent an extra hop. A wrong link sends it to the wrong table.

How many rules end up unlinked depends on how the source docs are written. Run the compiler on news:

python -m okf_compiler.cli news

The output (most of the list is cut):

news: 8 tables, 61 rules written to bundles\compiled\news
  rules not linked to any table (48):
    6: statusWeight
    7: genderFactor
    8: occupationFactor
    10: Premium Content Rule (PCR)
    11: Personalization Priority (PP)
    12: AB Testing Cohort Analysis (ABTCA)
    ... 42 more

In news, most rules are written in business language and never name a column. Rule 6, statusWeight, says 'Premium' = 2.0, 'Enterprise' = 3.0, but not which column holds the subscription tier. Others, such as rule 31, are built only from other rules' abbreviations. Name matching can't link these, so for those rules the agent has to find the right column itself, exactly as it would with the raw docs. The report tells you where your own documentation would need a human to add the missing link.

Open bundles/compiled/disaster/knowledge/resource-utilization-ratio.md. It should look like this:

---
type: Calculation
title: Resource Utilization Ratio (RUR)
description: Measures how effectively hub capacity is being used relative to available resources
tags:
- disaster
generated:
  by: okf_compiler/0.1
  at: '2026-10-03T00:11:39+05:30'
sources:
- id: kb
  resource: https://huggingface.co/datasets/birdsql/livesqlbench-base-lite/blob/main/disaster/disaster_kb.jsonl
  title: disaster business rules (LiveSQLBench), rule 10
---

# Definition

RUR = \frac{hubutilpct}{100} \times \frac{storecapm3}{storeavailm3 + 1}

# Columns used

* [distributionhubs](/tables/distributionhubs.md): `hubutilpct`, `storecapm3`, `storeavailm3`

# Used by

* [Financial Vulnerability Zone](/knowledge/financial-vulnerability-zone.md)
* [Resource Utilization Classification](/knowledge/resource-utilization-classification.md)

Generate the index files and validate

The bundle now has 64 concept files. An agent can't open all of them for every question, so it needs a menu first: an index.md in each folder that lists every concept with its one-line description. The agent reads the menu, then opens only what it needs.

Every line in an index comes from a concept file's frontmatter: its title, its file name and its description. So the index module doesn't need the raw data at all. It reads the concept files the compiler just wrote. That also means it works on any bundle folder, including one you edit by hand.

Create okf_compiler/indexes.py:

"""Write the index files and the log for a bundle."""

from datetime import date
from pathlib import Path

import yaml

from okf_compiler.concept import COMPILER

FOLDERS = {
    "tables": "PostgreSQL tables, with column meanings, JSON fields and joins.",
    "knowledge": "Business rules: calculations, definitions and value illustrations.",
}


def read_frontmatter(path: Path) -> dict:
    text = path.read_text(encoding="utf-8")
    return yaml.safe_load(text.split("---", 2)[1])


def write_folder_index(folder: Path) -> int:
    groups: dict[str, list[str]] = {}
    for path in sorted(folder.glob("*.md")):
        if path.name == "index.md":
            continue
        meta = read_frontmatter(path)
        line = f"* [{meta['title']}]({path.name}) - {meta['description']}"
        groups.setdefault(meta["type"], []).append(line)

    parts = []
    for concept_type, lines in sorted(groups.items()):
        parts += [f"# {concept_type}", "", *lines, ""]
    (folder / "index.md").write_text("\n".join(parts), encoding="utf-8")
    return sum(len(lines) for lines in groups.values())


def write_indexes(bundle: Path, database: str) -> None:
    counts = {name: write_folder_index(bundle / name) for name in FOLDERS}

    root = ["---", 'okf_version: "0.2"', "---", "", f"# {database} database", ""]
    for name, text in FOLDERS.items():
        root.append(f"* [{name.capitalize()}]({name}/index.md) - {text}")
    (bundle / "index.md").write_text("\n".join(root) + "\n", encoding="utf-8")

    log = [
        "# Update Log",
        "",
        f"## {date.today().isoformat()}",
        f"* **Creation**: Compiled {counts['tables']} tables and {counts['knowledge']} "
        f"business rules from LiveSQLBench `{database}` with `{COMPILER}`.",
    ]
    (bundle / "log.md").write_text("\n".join(log) + "\n", encoding="utf-8")

Notice three things:

Each folder index is grouped by type. In knowledge/index.md, calculations, business rules and value illustrations each get their own heading, so an agent looking for a formula can skip the definitions.
Only the root index has frontmatter, and only okf_version. OKF allows no other frontmatter in index files.
The log has a single entry. The compiler rebuilds the bundle from scratch on every run, so the log records one thing: which compiler version built it from which source, and when.

The final command

Replace okf_compiler/cli.py one last time. It now writes the indexes, and it accepts all to compile every database:

"""Compile one LiveSQLBench database into an OKF bundle."""

import shutil
import sys
from pathlib import Path

from okf_compiler.indexes import write_indexes
from okf_compiler.readers import add_column_meanings, read_rules, read_schema
from okf_compiler.rules import find_tables, write_rules
from okf_compiler.tables import write_table

DATA = Path("data/livesqlbench-base-lite")
OUT = Path("bundles/compiled")


def compile_database(name: str) -> None:
    folder = DATA / name
    tables = read_schema(folder / f"{name}_schema.txt")
    missing = add_column_meanings(tables, folder / f"{name}_column_meaning_base.json")
    rules = read_rules(folder / f"{name}_kb.jsonl")

    bundle = OUT / name
    shutil.rmtree(bundle, ignore_errors=True)
    related = write_rules(rules, tables, name, bundle)
    for table in tables:
        write_table(table, name, bundle, related[table.name])
    write_indexes(bundle, name)

    unlinked = [r for r in rules if not find_tables(r, tables)]
    print(f"{name}: {len(tables)} tables, {len(rules)} rules written to {bundle}")
    if missing:
        print(f"  columns without a description: {', '.join(missing)}")
    if unlinked:
        print(f"  rules not linked to any table ({len(unlinked)}):")
        for rule in unlinked:
            print(f"    {rule.id}: {rule.name}")


if __name__ == "__main__":
    if sys.argv[1] == "all":
        for folder in sorted(DATA.iterdir()):
            if (folder / f"{folder.name}_schema.txt").exists():
                compile_database(folder.name)
    else:
        compile_database(sys.argv[1])

Compile disaster again:

python -m okf_compiler.cli disaster

The output is the same report as before. The difference is in the bundle: it now has an index in each folder. Open bundles/compiled/disaster/knowledge/index.md. It should look like this (each group cut short):

# Business Rule

* [Community Resilience Builder](community-resilience-builder.md) - Identifies operations that strengthen local community capacity
...
* [Vulnerable Population Hotspot](vulnerable-population-hotspot.md) - Identifies areas with highly vulnerable populations requiring priority attention

# Calculation

* [available storage percentage](available-storage-percentage.md) - Calculates what proportion of total storage capacity is currently available
...
* [Supply Chain Sustainability Index (SCSI)](supply-chain-sustainability-index.md) - Assesses the environmental sustainability of the disaster supply chain

# Value Illustration

* [coordeffectlvl](coordeffectlvl.md) - Illustrates the quality of coordination between responding agencies
...
* [staffingProfile.readiness.ppe_status](staffingprofile-readiness-ppe-status.md) - Illustrates the availability of Personal Protective Equipment

Validate

Now check the bundle against the OKF rules with the validator from okf-skills, pinned to commit 8e31878 (download steps in the tutorial guide):

python tools/okf_validate.py bundles/compiled/disaster

The output:

OKF v0.2 conformance — bundles\compiled\disaster
  concepts: 64   index.md: 3   log.md: 1
  ✓ conformant — no issues

All 64 concepts, the 3 index files and the log are conformant, with no warnings. Every link points to a file that exists, because the compiler only writes links to tables and rules it has just generated.

All 18 databases

The same command works for every database in Base-Lite:

python -m okf_compiler.cli all

It prints one report per database (most of it cut):

alien: 11 tables, 56 rules written to bundles\compiled\alien
  rules not linked to any table (10):
    40: High-Confidence Technosignature
    41: Habitable Zone Transmission
    ... 8 more
... 16 more databases
virtual: 14 tables, 55 rules written to bundles\compiled\virtual
  rules not linked to any table (15):
    12: Content Creation Impact Score (CCIS)
    ... 14 more

Then validate them all. On macOS or Linux:

for db in bundles/compiled/*/; do python tools/okf_validate.py "$db" | tail -1; done

On Windows (PowerShell):

Get-ChildItem bundles/compiled -Directory | ForEach-Object { python tools/okf_validate.py $_.FullName | Select-Object -Last 1 }

Each bundle prints its last line, so you should see 18 of these:

  ✓ conformant — no issues
  ✓ conformant — no issues
  ✓ conformant — no issues
  ...

Seeing Γ£ô instead of ✓ in PowerShell? That's only the console's text encoding, not a problem with the bundles. Run [Console]::OutputEncoding = [Text.Encoding]::UTF8 once, then run the loop again.

Eighteen databases, eighteen conformant bundles.

What we built, and what we learned

We now have a compiler that turns a database's raw documentation into an OKF bundle with one command:

One concept file per table, with every column's meaning, its JSON fields and its joins.
One concept file per business rule, linked to the rules it depends on, the rules that use it, and the tables whose columns it names.
An index in every folder, so an agent can read a short menu before opening anything.
A report of what's missing: columns without a description, and rules the compiler couldn't link to a table.

Three lessons from running it on real documentation:

A compiler should report gaps, not fill them. It can't invent a description for patients.clinleadref, and it shouldn't guess which table a rule belongs to. A missing link costs the agent one extra hop; a wrong link sends it to the wrong table.
How well the links come out depends on how the docs are written. In disaster, 50 of 54 rules name their columns, so they link on their own. In news, 48 of 61 don't. The compiler can only link what the source spells out, and its report shows exactly where a human would need to add the missing link.
Keep generated knowledge honest. Every file says which compiler wrote it, from which source file and when, and none of them claims to be verified. Anyone reading the bundle can tell it was generated, not reviewed.

What's next

The bundle is ready, but nothing reads it yet. In the next part, we build the SQL agent on an open model, point it at the disaster bundle, and ask it questions twice: once browsing the bundle, and once with all the raw documentation pasted into its prompt. Then we compare what it read, how many tokens it used, and whether its SQL was right.

The full code for this part is in the okf-sql-knowledge repository.

Sources

LiveSQLBench: the text-to-SQL benchmark the disaster database comes from.
LiveSQLBench Base-Lite dataset on Hugging Face: the raw schema, column descriptions and business rules for all 18 databases.
LiveSQLBench PostgreSQL image: the database the agent will query.
okf-skills: the OKF validator used in this part (commit 8e31878).
Your agent's knowledge needs a file format: meet OKF: the introduction to OKF and its frontmatter fields.
The previous part: writing three disaster concept files by hand.
Tutorial guide: project setup, Docker and validator download steps.

Build a Knowledge Layer for SQL Agents with OKF (Part 2)

Lakhan Malviya — Sat, 03 Oct 2026 04:22:07 +0000

You connect an AI agent to your shop's database and ask a simple question: "How many active customers do we have?"

The agent looks at the tables:

customers (customer_id, name, status, created_at)
orders    (order_id, customer_id, amount, paid, ordered_at)

It spots a status column with the value 'active', writes a query, and gives you a number in two seconds:

SELECT COUNT(*) FROM customers WHERE status = 'active';

The number is wrong.

In your company, status = 'active' only means the account hasn't been closed. When finance says "active customer", they mean someone who placed a paid order in the last 90 days. Everyone on the data team knows this. It's written in a wiki page and built into the revenue dashboard. But it's not in the database, so the agent never saw it.

SELECT COUNT(DISTINCT customer_id)
FROM orders
WHERE paid = TRUE
  AND ordered_at >= CURRENT_DATE - INTERVAL '90 days';

This is the real problem with SQL agents. The database stores your data, but not the rules for reading it. Those rules live in wikis, dashboards, dbt models and people's heads. A new analyst picks them up by asking around. An agent can only read what you give it, and when a rule is missing, it guesses. The guess runs without errors, so nobody notices.

Is this a real problem?

It's fair to ask whether the shop example is realistic or just made up to prove a point. A public benchmark puts agents in exactly this situation.

LiveSQLBench tests how well AI agents turn questions into SQL. It contains 18 databases on different topics, from disaster response to crypto trading. Each database comes with three files:

the schema: tables and columns,
a description of every column,
a separate file of business rules.

The questions use business terms, and many of those terms are defined only in the rules file. It's the same situation as our shop: the database holds the data, and the definitions live somewhere else.

Here's one case from the disaster-response database. Two of its tables look like this (only the relevant columns are shown):

operations                (opsregistry, emerglevel, ...)
coordinationandevaluation (coordopsref, safetyranking, secincidentcount, ...)

Ask an agent for "high-risk operations", and it will spot the safetyranking column, which has a value called High Risk, and filter on it, just as our shop agent filtered on status. But the rules file defines a high-risk operation as one where emerglevel is Red or Black, safetyranking is High Risk, and secincidentcount is more than 50. That's three conditions across two tables. The obvious guess catches one.

It's a hard test. At the time of writing, the best system on the LiveSQLBench leaderboard answers 48% of its tasks correctly.

What we'll build

So how do you give an agent the rules it's missing? Teams usually try one of two things.

The first is to paste everything into the prompt: every table, every column description, every rule. That works for a small database. But you pay for all those tokens on every question, and for a large database the documents don't fit in the model's context window at all.

The second is to let the agent guess from the schema alone. It's cheap, and it gives you the wrong answers we just saw.

This series builds a third option: a knowledge layer. We take the schema, the column descriptions and the business rules, and store them as an OKF bundle: a folder of small markdown files called concept files, one per table and one per business rule, with an index file in each folder.

An agent can read an OKF bundle in more than one way. It can search the files, follow the links between them like a graph, or walk through the index files. We start with the simplest, called progressive disclosure: the agent reads the top index first, opens only the concept files the question needs, and then writes SQL. OKF's index files exist for exactly this, although they're optional and other readers can work without them.

Raw docs: schema, column descriptions, business rules
        │   compiler
        ▼
OKF bundle: concept files + index files
        │   agent tools (plain Python, then MCP)
        ▼
Agent opens what it needs → writes SQL
        │   evaluation harness
        ▼
Score on LiveSQLBench

The series has four parts. Each one ends with something you can run on your laptop:

The problem and your first bundle (this post). Set up the project, explore one LiveSQLBench database, write a small OKF bundle by hand, and validate it.
The compiler. A Python script that turns a database's raw files into a full OKF bundle automatically, for all 18 databases.
The agent. Run the benchmark database locally in Docker, then build an agent that reads the bundle through progressive disclosure and writes SQL: first in plain Python, then over MCP (Model Context Protocol), so any MCP-compatible agent can use the same knowledge.
The honest test. Run the same benchmark questions four ways with open models: schema only, everything in the prompt, progressive disclosure over the bundle, and search over the same bundle. Then compare accuracy, tokens and cost, whatever the results show.

Set up the project

You need Python 3.11 or newer (the OKF validator we use later in this post requires it) and Git. Step-by-step setup for macOS and Windows is in the tutorial guide in the project repo. (If you'd rather run the finished code than build it yourself, follow the README instead.)

Following the guide, you create a project folder with this structure:

okf-sql-knowledge/
├── data/       raw LiveSQLBench files
└── bundles/    the OKF bundles we write

Both folders start empty. We fill data/ next and bundles/ later in this post.

Get the data

LiveSQLBench comes in several releases. We use Base-Lite, the smallest one: 18 databases and 270 questions. Download it into data/ as shown in the tutorial guide. It's only a few megabytes.

Here's what you get:

data/livesqlbench-base-lite/
├── README.md
├── livesqlbench_data.jsonl
├── alien/
├── archeology/
├── credit/
├── disaster/
├── ...              (18 database folders in total)
└── virtual/

There are two kinds of things here:

One folder per database (alien, credit, disaster, …). Each folder describes one database: its tables, its columns and its business rules.
livesqlbench_data.jsonl: the questions, one per line, 15 for each database.

Here's the first question for the disaster database, shortened to the fields that matter now:

{
  "instance_id": "disaster_1",
  "selected_database": "disaster",
  "query": "I need to analyze all distribution hubs based on their Resource Utilization Ratio. Please show the hub registry ID, the calculated RUR value, and their Resource Utilization Classification. Sort the results by RUR from highest to lowest.",
  "sol_sql": []
}

Notice two things.

First, the question asks for a "Resource Utilization Ratio". No column has that name. Its formula lives in the business rules, which is exactly the problem from the start of this post.

Second, sol_sql, the correct answer, is empty. The benchmark's authors hold back the answers so automated web crawlers can't collect them. You can request them by email, as the dataset page explains. We won't need them until we test the agent.

From here on, we work with the disaster database. Its folder has three files, each around 20 KB:

data/livesqlbench-base-lite/disaster/
├── disaster_schema.txt
├── disaster_column_meaning_base.json
└── disaster_kb.jsonl

Read the three raw files

We'll follow the question we just saw. To answer it, an agent needs the distribution hubs table, the formula for the Resource Utilization Ratio, and the rule that turns that ratio into a classification. Each piece lives in a different file. (For a full tour of every file and field, see data/README.md in the repo.)

The schema: `disaster_schema.txt`

Plain text, one CREATE TABLE statement per table, each followed by three sample rows. The table we need has 11 columns:

CREATE TABLE "distributionhubs" (
hubregistry character varying NOT NULL,
disteventref character varying NULL,
hubcaptons numeric NULL,
hubutilpct numeric NULL,
storecapm3 numeric NULL,
storeavailm3 numeric NULL,
coldstorecapm3 numeric NULL,
coldstoretempc numeric NULL,
warehousestate USER-DEFINED NULL,
invaccpct numeric NULL,
stockturnrate numeric NULL,
    PRIMARY KEY (hubregistry),
    FOREIGN KEY (disteventref) REFERENCES disasterevents(distregistry)
);

The schema gives names, types and joins. It doesn't say what hubutilpct or storeavailm3 mean.

The column descriptions: `disaster_column_meaning_base.json`

One sentence per column, keyed by database|table|column:

"disaster|distributionhubs|hubutilpct": "A DECIMAL(7,3) showing the percentage of hub capacity currently utilized (e.g., 85.300)."

Now hubutilpct makes sense: it's the percentage of the hub's capacity in use.

The business rules: `disaster_kb.jsonl`

This is the file our agent was missing in the opening. It holds 54 rules, one JSON object per line. Here are the two our question needs, formatted for reading:

{
  "id": 10,
  "knowledge": "Resource Utilization Ratio (RUR)",
  "description": "Measures how effectively hub capacity is being used relative to available resources",
  "definition": "RUR = \\frac{hubutilpct}{100} \\times \\frac{storecapm3}{storeavailm3 + 1}",
  "type": "calculation_knowledge",
  "children_knowledge": -1
}

{
  "id": 50,
  "knowledge": "Resource Utilization Classification",
  "description": "Categorizes distribution hubs based on their Resource Utilization Ratio (RUR) values",
  "definition": "High Utilization (RUR > 5) indicates potentially overloaded hubs that may need resource expansion; Moderate Utilization (2 ≤ RUR ≤ 5) represents optimal resource usage balance; Low Utilization (RUR < 2) indicates underutilized hubs with potential efficiency gains through resource reallocation",
  "type": "domain_knowledge",
  "children_knowledge": [10]
}

Three fields matter most for what we build next:

type says what kind of rule it is: a formula (calculation_knowledge), a business definition (domain_knowledge), or an explanation of a column's values (value_illustration).
definition is the rule itself. Rule 10 is a LaTeX formula: RUR = (hubutilpct / 100) × (storecapm3 / (storeavailm3 + 1)).
children_knowledge lists the rules this one depends on. Rule 50 lists [10], because you can't classify a hub without first computing its RUR.

Notice what's missing. Rule 10 names three columns but not their table. To learn that they belong to distributionhubs, you have to go back to the schema.

What we learned

No file is enough on its own:

File	Answers	Example
Schema	What tables and columns exist, and how they join	`distributionhubs` has `hubutilpct`
Column descriptions	What each column means	`hubutilpct` is the percentage of capacity in use
Business rules	What the business terms mean	RUR is a formula over three of those columns

Write your first concept files by hand

Now we turn those pieces into OKF. A concept file is a markdown file with a block of YAML at the top, called frontmatter, and a normal markdown body below it. (What each frontmatter field means is covered in the introduction to OKF; here we focus on using them.)

We'll write three concept files: one for the table, and one for each rule. Your project will look like this:

okf-sql-knowledge/
├── data/
│   └── livesqlbench-base-lite/
├── bundles/
│   └── disaster/
│       ├── tables/
│       │   └── distributionhubs.md
│       └── knowledge/
│           ├── resource-utilization-ratio.md
│           └── resource-utilization-classification.md
├── requirements.txt
└── README.md

Tables go in tables/, rules go in knowledge/. OKF doesn't require any particular folder names; this split simply makes the bundle easy to browse.

The table concept

Create bundles/disaster/tables/distributionhubs.md:

---
type: PostgreSQL Table
title: distributionhubs
description: One row per disaster-relief distribution hub, with its capacity, storage space, cold storage, warehouse condition and inventory measures.
tags: [disaster, logistics]
generated: { by: human:your-name, at: 2026-10-02T17:00:00+05:30 }
sources:
  - id: schema
    resource: https://huggingface.co/datasets/birdsql/livesqlbench-base-lite/blob/main/disaster/disaster_schema.txt
    title: disaster schema (LiveSQLBench)
  - id: column-meanings
    resource: https://huggingface.co/datasets/birdsql/livesqlbench-base-lite/blob/main/disaster/disaster_column_meaning_base.json
    title: disaster column descriptions (LiveSQLBench)
---

# Schema

| Column | Type | Meaning |
|---|---|---|
| `hubregistry` | varchar, primary key | Hub ID, for example `HUB_HS0I`. |
| `disteventref` | varchar | The disaster event this hub serves. |
| `hubcaptons` | numeric | Maximum capacity of the hub, in tons. |
| `hubutilpct` | numeric | Percentage of the hub's capacity currently in use. |
| `storecapm3` | numeric | Total storage capacity, in cubic meters. |
| `storeavailm3` | numeric | Storage still available, in cubic meters. |
| `coldstorecapm3` | numeric | Cold-storage capacity, in cubic meters. |
| `coldstoretempc` | numeric | Temperature of the cold storage, in °C. |
| `warehousestate` | enum | Warehouse condition: `Fair`, `Excellent`, `Good` or `Poor`. |
| `invaccpct` | numeric | Inventory accuracy, as a percentage. |
| `stockturnrate` | numeric | How often the inventory is turned over in a period. |

# Joins

`disteventref` references `distregistry` in [disasterevents](/tables/disasterevents.md).

# Related knowledge

* [Resource Utilization Ratio (RUR)](/knowledge/resource-utilization-ratio.md) is computed from `hubutilpct`, `storecapm3` and `storeavailm3`.

Replace your-name with your own name, and the date with the current time.

This one file now holds what used to be split across two raw files: the columns and types from the schema, and their meanings from the column descriptions. A few things to notice:

type is the only required field. OKF has no fixed list of types, so we pick a name that says what the concept is.
description is the line an agent sees before it opens the file. We'll use it in the index files in the next step, so it should say clearly what's in the table.
generated records who wrote the file and when, and sources points back to the raw files it came from. human: marks a person as the author.
Links use paths that start with /. They point from the root of the bundle, so they stay correct wherever the file sits.
The link to disasterevents points to a file we haven't written. That's on purpose. OKF allows links to knowledge that doesn't exist yet, and we'll see how the validator treats it.

The rule concepts

Create bundles/disaster/knowledge/resource-utilization-ratio.md for rule 10:

---
type: Calculation
title: Resource Utilization Ratio (RUR)
description: Measures how effectively hub capacity is being used relative to available resources.
tags: [disaster, logistics]
generated: { by: human:your-name, at: 2026-10-02T17:00:00+05:30 }
sources:
  - id: kb
    resource: https://huggingface.co/datasets/birdsql/livesqlbench-base-lite/blob/main/disaster/disaster_kb.jsonl
    title: disaster business rules (LiveSQLBench), rule 10
---

# Definition

RUR = (hubutilpct / 100) × (storecapm3 / (storeavailm3 + 1))

# Columns used

All three columns are in [distributionhubs](/tables/distributionhubs.md): `hubutilpct`, `storecapm3` and `storeavailm3`.

# Used by

* [Resource Utilization Classification](/knowledge/resource-utilization-classification.md)

And bundles/disaster/knowledge/resource-utilization-classification.md for rule 50:

---
type: Business Rule
title: Resource Utilization Classification
description: Categorizes distribution hubs based on their Resource Utilization Ratio (RUR) values.
tags: [disaster, logistics]
generated: { by: human:your-name, at: 2026-10-02T17:00:00+05:30 }
sources:
  - id: kb
    resource: https://huggingface.co/datasets/birdsql/livesqlbench-base-lite/blob/main/disaster/disaster_kb.jsonl
    title: disaster business rules (LiveSQLBench), rule 50
---

# Definition

| Class | Condition | Meaning |
|---|---|---|
| High Utilization | RUR > 5 | Possibly overloaded; may need more resources. |
| Moderate Utilization | 2 ≤ RUR ≤ 5 | Balanced use of resources. |
| Low Utilization | RUR < 2 | Underused; resources could be moved elsewhere. |

# Depends on

* [Resource Utilization Ratio (RUR)](/knowledge/resource-utilization-ratio.md), which must be computed first.

What changed compared with the raw rules:

Each rule type got a readable name. calculation_knowledge became Calculation, and domain_knowledge became Business Rule.
The formula is plain math instead of LaTeX, and the classification is a table instead of one long sentence. The content is the same; it's just easier to read.
The rule now says where its columns live. The raw rule 10 named three columns but not their table. The "Columns used" section links to distributionhubs, so an agent no longer has to search the schema.
The dependency is a link in both directions. "Depends on" in rule 50 replaces children_knowledge: [10]. "Used by" in rule 10 is the reverse link, so an agent that starts from either rule can find the other.

Together, these three files hold everything the disaster_1 question needs. But an agent still has to find them. That's the job of index files, which we add next.

Add index files and a log

Our bundle has three concept files. The full disaster database will have 64: 10 tables and 54 rules. An agent can't open all of them for every question; that would be no better than pasting everything into the prompt.

It needs a menu first. In OKF, that menu is a file called index.md. Each folder can have one, and it lists what's in that folder, one line per item, with a short description. The agent reads the menu, then opens only what it needs.

The root index

Create bundles/disaster/index.md:

---
okf_version: "0.2"
---

# disaster database

* [Tables](tables/index.md) - PostgreSQL tables of the disaster-response database, with column meanings and joins.
* [Knowledge](knowledge/index.md) - Business rules: formulas and definitions used in questions about this database.

This is the agent's front door. It points to the index of each folder and says what that folder holds.

The block at the top is optional. It declares which OKF version the bundle follows, which is useful because the spec can change between versions. It's the only frontmatter an index file may have, and only in the index at the root of the bundle. The other index files have no frontmatter at all.

One index per folder

Create bundles/disaster/tables/index.md:

# Tables

* [distributionhubs](distributionhubs.md) - One row per disaster-relief distribution hub, with its capacity, storage space, cold storage, warehouse condition and inventory measures.

And bundles/disaster/knowledge/index.md, grouped by rule type:

# Calculations

* [Resource Utilization Ratio (RUR)](resource-utilization-ratio.md) - Measures how effectively hub capacity is being used relative to available resources.

# Business rules

* [Resource Utilization Classification](resource-utilization-classification.md) - Categorizes distribution hubs based on their Resource Utilization Ratio (RUR) values.

Each line is the concept's title, a link to its file, and its description, copied from the frontmatter. This is why the description matters so much: in the index, it's the only thing the agent sees before deciding whether to open a file.

What the indexes make possible

With these files in place, everything the disaster_1 question needs can be reached from the root index in a few hops: from the knowledge index to the two rules, and from the rules to the table they use. Whether a real agent actually takes that path is something we'll check later in the series, when we build the agent and log what it opens.

The three index files together are under 1 KB. For comparison, the three raw files for this database total about 63 KB.

There's a catch. Every time you add a concept, you also have to add its line to the right index, by hand. With three files that's easy. With 64, it's tedious and easy to get wrong. We'll automate it in the next part.

The log

Finally, create bundles/disaster/log.md:

# Update Log

## 2026-10-02
* **Creation**: Added the [distributionhubs](/tables/distributionhubs.md) table, the [Resource Utilization Ratio](/knowledge/resource-utilization-ratio.md) and the [Resource Utilization Classification](/knowledge/resource-utilization-classification.md), written by hand from LiveSQLBench `disaster`.

The log is a short history of changes to the bundle, newest first, grouped under dates. It has no frontmatter. Humans read it to see what changed and when; an agent can read it too, but it doesn't need it to answer questions.

Your project now looks like this:

okf-sql-knowledge/
├── data/
│   └── livesqlbench-base-lite/
├── bundles/
│   └── disaster/
│       ├── index.md
│       ├── log.md
│       ├── tables/
│       │   ├── index.md
│       │   └── distributionhubs.md
│       └── knowledge/
│           ├── index.md
│           ├── resource-utilization-ratio.md
│           └── resource-utilization-classification.md
├── requirements.txt
└── README.md

Validate the bundle, then break it on purpose

Before any agent reads a bundle, whoever wrote it should check that it follows the OKF rules. A typo in the frontmatter is easy to miss by eye, and an agent won't tell you about it.

We'll use the validator from okf-skills, an open-source (MIT) toolkit for OKF. It's a single Python file. Download it as shown in the tutorial guide; the link is pinned to a fixed version, so you see the same output as below. Then run it from the project folder, with your virtual environment active:

python tools/okf_validate.py bundles/disaster

OKF v0.2 conformance — bundles/disaster
  concepts: 3   index.md: 3   log.md: 1
  ! warn   tables/distributionhubs.md: cross-link target not found: `/tables/disasterevents.md` (tolerated under §6.1)
  ✓ conformant (1 warning(s))

The bundle is conformant: it follows the OKF v0.2 rules. There's one warning, for the link to disasterevents that we left in on purpose.

That's the key difference between the two kinds of findings:

An error breaks a rule the spec requires. The bundle is not valid until you fix it.
A warning points at something worth a look, but the spec tells readers of a bundle to cope with it. A link to a file that doesn't exist yet is fine: it may simply be knowledge nobody has written.

The best way to learn the rules is to break them, so let's do that twice.

Break 1: remove the `type`

Open knowledge/resource-utilization-ratio.md and delete the line type: Calculation. Run the validator again:

OKF v0.2 conformance — bundles/disaster
  concepts: 3   index.md: 3   log.md: 1
  ✗ ERROR  knowledge/resource-utilization-ratio.md: §11.2 missing or empty required `type` field
  ! warn   tables/distributionhubs.md: cross-link target not found: `/tables/disasterevents.md` (tolerated under §6.1)
  ✗ non-conformant (1 error(s))

Now it's an error, and the whole bundle is non-conformant. type is the one field every concept must have; everything else in the frontmatter is optional. Put the line back before moving on.

Break 2: add frontmatter to a folder index

Earlier we said that only the root index.md may have frontmatter. Let's test that. Add this block to the very top of tables/index.md:

---
title: Tables
---

Run the validator:

OKF v0.2 conformance — bundles/disaster
  concepts: 3   index.md: 3   log.md: 1
  ! warn   tables/index.md: §8 index.md should contain no frontmatter
  ! warn   tables/distributionhubs.md: cross-link target not found: `/tables/disasterevents.md` (tolerated under §6.1)
  ✓ conformant (2 warning(s))

It's only a warning, so the bundle still counts as conformant. But the OKF spec is clear that folder index files have no frontmatter, so remove the block again. If you're ever unsure about a format rule, check the official OKF spec.

Run the validator once more. When you see just the one expected warning, your first OKF bundle is done.

What's next

You now have a small, valid OKF bundle: one table and two rules, linked to each other and listed in index files. Along the way, we saw three things:

The schema isn't enough. Business definitions live elsewhere, and an agent without them guesses.
Links and indexes make knowledge findable. An agent can start from one small file and reach what a question needs.
A clear description matters most. It's what an agent sees before opening a file.

Three files by hand was easy. The disaster database alone needs 64, and all 18 databases need over a thousand. In the next part, we write a Python script that generates them all, with every link and index, and validates the result.

The code for this series is in the okf-sql-knowledge repository.

Sources

Open Knowledge Format (OKF) specification, v0.2 — Google Cloud. Format rules used throughout: frontmatter and type (§4), links (§6), index files (§8), log files (§9), conformance (§11).
Your Agents' Knowledge Needs a File Format. Meet OKF — the introduction to OKF and its frontmatter fields.
LiveSQLBench — benchmark website and leaderboard (48% best success rate, checked October 2026).
LiveSQLBench on GitHub — BIRD team at HKU and Google Cloud. Code, evaluation and database setup.
LiveSQLBench-Base-Lite dataset — the 18 databases and 270 questions used in this post, and the note on withheld answers.
okf-skills — Marco Boffo, MIT license. The OKF validator (okf_validate.py, commit 8e31878).
okf-sql-knowledge — the code and guides for this series.

DEV Community: Lakhan Malviya

Your Schema Isn't Enough: Build a SQL Agent That Reads Business Knowledge First (OKF Series, Part 4)

Set up the project for the agent

Connect to a model and get a test answer

The settings file

One function for every model call

Run the test

Build the tools: read a bundle file, submit an answer

Try the tool yourself

Build the agent loop

When the request isn't a data question

Record every step

Start the database with a read-only login for the agent

Run SQL safely

Give the agent the tool

Put it all behind one command

Ask your first question

Ask the same question without the bundle

What's next

Sources

Turn Database Docs into Agent-Ready Knowledge (OKF Series, Part 3)

Set up the project and the database

The project layout

The database

Read the raw files into Python

Check what was read

Generate the table concepts

A helper for every concept file

The table module

The command

Generate the rule concepts and their links

Update the command

Generate the index files and validate

The final command

Validate

All 18 databases

What we built, and what we learned

What's next

Sources

Build a Knowledge Layer for SQL Agents with OKF (Part 2)

Is this a real problem?

What we'll build

Set up the project

Get the data

Read the three raw files

The schema: disaster_schema.txt

The column descriptions: disaster_column_meaning_base.json

The business rules: disaster_kb.jsonl

What we learned

Write your first concept files by hand

The table concept

The rule concepts

Add index files and a log

The root index

One index per folder

What the indexes make possible

The log

Validate the bundle, then break it on purpose

Break 1: remove the type

Break 2: add frontmatter to a folder index

What's next

Sources

The schema: `disaster_schema.txt`

The column descriptions: `disaster_column_meaning_base.json`

The business rules: `disaster_kb.jsonl`

Break 1: remove the `type`