A Free-Server Regression Job for LLM-Generated SQL

LLM-generated SQL breaks in ways that ordinary unit tests miss. A model can return a syntactically valid query against a schema it has never seen, then a week later return the same-sounding query with an extra join, a wrong date filter, or a column alias that breaks the reporting tool. Teams often respond by adding a paid CI job that provisions a database and runs every prompt through the latest model. That approach works, but it adds cost and setup time before the first useful signal arrives.

This tutorial builds a small nightly regression check that runs against a frozen SQLite fixture database. It uses the free model access and free server option available in MonkeyCode to keep the loop near zero cost. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harness has three parts: a fixture schema, a set of prompt-to-SQL cases with golden outputs, and a runner that calls the model endpoint, executes the generated query inside a rolled-back transaction, and compares a hash of the result columns and rows.

Why a frozen fixture matters

A regression check is only useful when the baseline is stable. A live development database changes constantly, so a failing SQL test could mean the model improved, the data shifted, or an analyst added a column. Those causes are hard to separate. A frozen SQLite fixture removes that variable. Because SQLite is file-based, the fixture can be stored in the repository and rebuilt deterministically before each run.

What the harness does

• Creates a schema with a few tables that mirror a typical reporting surface.
• Reads a JSONL file containing natural-language prompts and expected SQL.
• Sends each prompt to a model endpoint through a thin HTTP client.
• Executes the returned SQL inside a transaction and rolls it back so the fixture stays unchanged.
• Hashes the result columns and rows and compares them with a golden hash.
• Exits with a non-zero code when any case differs.

What the harness does not do

• It does not evaluate query performance.
• It does not verify business intent across arbitrary new questions.
• It does not replace a real database dialect test.

Prerequisites

• Python 3.11 or later.
• A MonkeyCode account with free model access and a free server option.
• The current model endpoint URL and token stored in environment variables.
• A small repository with the fixture schema, cases file, and runner script.

The code below is a template. Replace endpoint names, request shapes, and response parsing with the values shown in the current MonkeyCode API documentation.

Step 1: Create the fixture schema

Create schema.sql with a small reporting-like dataset.

CREATE TABLE customer (
  customer_id INTEGER PRIMARY KEY,
  signup_date TEXT NOT NULL,
  country TEXT NOT NULL
);

CREATE TABLE orders (
  order_id INTEGER PRIMARY KEY,
  customer_id INTEGER NOT NULL REFERENCES customer(customer_id),
  amount NUMERIC NOT NULL,
  order_date TEXT NOT NULL
);

INSERT INTO customer VALUES
  (1, '2025-01-10', 'US'),
  (2, '2025-02-14', 'CA'),
  (3, '2025-03-01', 'US');

INSERT INTO orders VALUES
  (101, 1, 120.00, '2025-06-01'),
  (102, 1, 80.50, '2025-06-15'),
  (103, 2, 200.00, '2025-06-20'),
  (104, 3, 15.00, '2025-07-01');

Step 2: Define prompt cases and golden hashes

Create cases.jsonl. Each line contains a prompt, the expected SQL, and a golden hash computed from the expected result columns and rows.

{"id":"us_revenue","prompt":"Find total revenue from US customers ordered by order date ascending","golden_sql":"SELECT SUM(o.amount) AS total_revenue FROM orders o JOIN customer c ON c.customer_id = o.customer_id WHERE c.country = 'US'","golden_hash":"replace-with-computed-hash"}
{"id":"customer_order_count","prompt":"Return each customer and their order count, including customers with no orders","golden_sql":"SELECT c.customer_id, COUNT(o.order_id) AS order_count FROM customer c LEFT JOIN orders o ON o.customer_id = c.customer_id GROUP BY c.customer_id ORDER BY c.customer_id","golden_hash":"replace-with-computed-hash"}

Step 3: Build the model client

Create generate_sql.py.

import os
import httpx

def generate_sql(prompt: str) -> str:
    resp = httpx.post(
        os.environ['MONKEYCODE_API_URL'],
        headers={'Authorization': 'Bearer ' + os.environ['MONKEYCODE_API_TOKEN']},
        json={
            'model': os.environ.get('MONKEYCODE_MODEL', 'free-default'),
            'messages': [{'role': 'user', 'content': prompt}],
        },
        timeout=120,
    )
    resp.raise_for_status()
    data = resp.json()
    return data['choices'][0]['message']['content'].strip()

The response parsing assumes an OpenAI-compatible choices[0].message.content shape. If the current MonkeyCode endpoint returns a different shape, adjust the client or wrap it with a small adapter rather than changing the rest of the harness.

Step 4: Execute and hash the result

Create runner.py.

import hashlib
import json
import sqlite3
import sys
from pathlib import Path

from generate_sql import generate_sql

def hash_result(columns, rows):
    payload = json.dumps(
        {'columns': columns, 'rows': rows}, sort_keys=True, default=str
    )
    return hashlib.sha256(payload.encode()).hexdigest()

def execute_read_only(db_path: str, sql_text: str):
    conn = sqlite3.connect(db_path)
    cur = conn.cursor()
    cur.execute('BEGIN')
    try:
        cur.execute(sql_text)
        rows = cur.fetchall()
        columns = [description[0] for description in cur.description]
        return columns, rows
    finally:
        conn.rollback()
        conn.close()

def load_cases(cases_path: str):
    cases = []
    for line in Path(cases_path).read_text().splitlines():
        if line.strip():
            cases.append(json.loads(line))
    return cases

def main():
    if len(sys.argv) != 3:
        print('usage: python runner.py <db_path> <cases_path>', file=sys.stderr)
        sys.exit(2)

    db_path, cases_path = sys.argv[1], sys.argv[2]
    failures = []

    for case in load_cases(cases_path):
        generated_sql = generate_sql(case['prompt'])
        columns, rows = execute_read_only(db_path, generated_sql)
        actual_hash = hash_result(columns, rows)

        if actual_hash != case['golden_hash']:
            failures.append({
                'id': case['id'],
                'expected_hash': case['golden_hash'],
                'actual_hash': actual_hash,
                'generated_sql': generated_sql,
            })

    if failures:
        print(json.dumps(failures, indent=2))
        sys.exit(1)

    print('all cases passed')

if __name__ == '__main__':
    main()

The runner does not compare golden_sql to the generated SQL. It compares the executed result hash, which tolerates semantically equivalent query rewrites while still catching output changes.

Step 5: Compute the golden hashes once

Before the nightly run starts, compute the expected hashes against the fixture database using the SQL stored in each case.

# compute_hashes.py
import json
import sys
from runner import execute_read_only, hash_result

def main():
    db_path, cases_path = sys.argv[1], sys.argv[2]
    for case in json.loads(open(cases_path).read()):
        columns, rows = execute_read_only(db_path, case['golden_sql'])
        case['golden_hash'] = hash_result(columns, rows)
        print(json.dumps(case))

if __name__ == '__main__':
    main()

Run it with:

python compute_hashes.py fixture.db cases.jsonl > cases.hashed.jsonl

Then move the hashed file back to cases.jsonl.

Step 6: Schedule the run on the free server

Because the harness is a single Python file and a SQLite fixture, it fits on a small free server. Store the following files in the same directory:

• schema.sql
• cases.jsonl
• generate_sql.py
• runner.py
• compute_hashes.py

Rebuild the fixture at the start of each run so the server does not depend on persistent local state.

sqlite3 fixture.db < schema.sql
python runner.py fixture.db cases.jsonl

A minimal cron entry can run it every night at 03:00.

0 3 * * * cd /path/to/sql-regression && sqlite3 fixture.db < schema.sql && python runner.py fixture.db cases.jsonl

Replace cd /path/to/sql-regression with the actual free server path. Keep the run quiet by appending > /tmp/sql-regression.log 2>&1 when that is supported.

Decision table: when this fits

Situation	Good fit?	Notes
Fixture has fewer than 10,000 rows	Yes	SQLite handles this quickly on a small server.
Prompts are mostly SELECT statements	Yes	The transaction rollback pattern expects read-only queries.
Queries include vendor-specific dialect features	No	SQLite will not validate Oracle, PostgreSQL, or SQL Server syntax.
You need a signal on every pull request	Possibly	A free server may not be fast enough or always available.
You expect private schema data	No	Keep sensitive schemas inside a controlled environment.
You want a production performance check	No	This harness checks output shape and values, not latency or plans.

Limitations

• Free model access and free server resources are limited. Keep the fixture and case list small.
• The model endpoint may return multiple SQL statements or explanatory text. Strip the text carefully or wrap the call to extract only the first SQL block.
• A hash comparison is strict. Adding a column, changing row order, or changing a number format will fail even if the answer is still useful.
• SQLite is not a substitute for the production database engine.
• The free server may be ephemeral. Keep all configuration in the repository and rebuild from schema.sql.

Who should not use this approach

• Teams that need dialect-specific validation against a real database engine.
• Teams with large, complex, or sensitive schemas.
• Teams that require sub-second model responses at high request volume.
• Teams that already have an expensive, maintained test environment and need more than a lightweight regression signal.

Closing note

A small frozen-fixture harness lowers the cost of noticing when a model's SQL output changes. It is deliberately narrow: one fixture, a few prompts, and a strict result hash. That narrowness makes it easy to reason about and cheap to run. The next step is to add cases for the queries that have caused the most incidents, not to model every possible question.