Synthetic Data Is the Missing Layer in Your SQL Eval Harness

#sql #ai #testing #dataprivacy

Last spring, a colleague's team shipped an AI-generated query that accidentally exposed row-level customer data from a supposedly anonymized copy of their production database. The anonymization had removed names, but the query's WHERE clause still matched on a combination of attributes that re-identified individuals. That incident taught me a hard rule: any dataset used to evaluate LLM-generated SQL must be synthetic, unless you can prove your real data is both safe and representative.

Most teams I talk to skip this step because they think synthetic data is too much work or too unrealistic. They are wrong on both counts. A free tier with a token allowance and a disposable server can turn synthetic data generation into a routine part of your eval harness, and it costs nothing but a few hours of setup.

In this post, I will argue that synthetic data is not a nice-to-have for SQL model evaluation; it is the foundation. I will show you a concrete pipeline that uses MonkeyCode's free model access and free server option to generate, load, and test against synthetic data, and I will explain where this approach breaks down.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Real Data Is a Broken Test Fixture

Real data carries the exact biases you want your model to unlearn, but it also carries legal and ethical baggage that no test suite should inherit. A production copy, even with columns renamed, still contains statistical fingerprints that can identify individuals, and a generated query that filters on those fingerprints becomes a privacy incident waiting to happen. Beyond compliance, real data is also a poor test oracle because it changes over time, so a regression that passes today may fail tomorrow for reasons unrelated to your model.

The Free Tier as a Synthetic Data Factory

MonkeyCode's free tier includes a 10 million token allowance and a server option that you can treat as a disposable sandbox. That combination is ideal for generating synthetic data, because you can prompt the model to write INSERT statements that match your schema, then execute them on a throwaway SQLite database. The server gives you a clean environment, and the token allowance lets you iterate on the generation prompt without watching a meter.

Here is a minimal Python script that builds a synthetic fixture from a schema definition:

# synthetic_fixture.py — minimal example, not production code
import sqlite3
import random
import string

SCHEMA = '''
CREATE TABLE users (
    id INTEGER PRIMARY KEY,
    email TEXT NOT NULL,
    country TEXT NOT NULL,
    age INTEGER
);
'''

def fake_email():
    return ''.join(random.choices(string.ascii_lowercase, k=8)) + '@example.com'

def build_fixture(path: str, rows: int = 1000) -> None:
    conn = sqlite3.connect(path)
    conn.executescript(SCHEMA)
    countries = ['US', 'GB', 'DE', 'FR', 'JP']
    for i in range(rows):
        conn.execute(
            'INSERT INTO users (email, country, age) VALUES (?, ?, ?)',
            (fake_email(), random.choice(countries), random.randint(18, 80))
        )
    conn.commit()
    conn.close()

if __name__ == '__main__':
    build_fixture('fixture.sqlite3')

This script is deliberately simple, but it already gives you three properties that real data lacks: controlled cardinality, known distributions, and no privacy risk. You can extend it with edge cases like NULLs, duplicate emails, or extreme ages, and you can regenerate it deterministically with a fixed random seed.

Evaluating Generated SQL on Synthetic Data

Once the fixture exists, you can use the free model access to generate SQL for a natural-language task, then run that SQL against the synthetic database and score the result. The key is to treat the synthetic data as an oracle: you know the exact row counts and distributions, so any deviation from the expected output is a real bug in the generated query.

# eval_generated_sql.py — minimal example, not production code
import sqlite3
import json
import sys

def evaluate(sql_path: str, fixture: str) -> dict:
    conn = sqlite3.connect(fixture)
    conn.execute('PRAGMA query_only = ON')
    cur = conn.cursor()
    sql = open(sql_path).read()
    try:
        cur.execute(sql)
        rows = cur.fetchall()
        return {'ok': True, 'row_count': len(rows)}
    except Exception as exc:
        return {'ok': False, 'error': str(exc)}
    finally:
        conn.close()

if __name__ == '__main__':
    print(json.dumps(evaluate(sys.argv[1], 'fixture.sqlite3')))

The read-only pragma ensures that a generated DELETE or UPDATE fails loudly instead of corrupting your fixture. That loud failure is exactly what you want from a test environment, and it is why the free server is the right place to run this loop.

Limitations and Who Should Skip This

Synthetic data cannot reproduce the performance characteristics of a production database, so it is useless for benchmarking query execution times. It also cannot capture the long-tail of real-world data quality issues, such as inconsistent encodings or subtle cross-column correlations. If your goal is to test index behavior or query plans under realistic load, you need a different approach. But if your goal is to verify that a model-generated query returns the correct logical result, synthetic data is not just acceptable; it is superior to any anonymized copy.

The Bottom Line

A free tier with a token allowance and a disposable server is not a production gift; it is a test infrastructure gift. The most valuable thing you can build with it is a synthetic data pipeline that makes your SQL eval harness reproducible, private, and deterministic. Start with a simple schema, generate a few hundred rows, and see how your favorite model performs when the answer is known in advance.

If you already have an eval harness, add a synthetic fixture to it this weekend. If you do not, build one with the script above and let the free tier do the heavy lifting.

DEV Community