Building a Zero-Budget Eval Harness for LLM-Assisted SQL Generation

Most data teams I read about have the same quiet problem: they use an LLM to help write SQL, it works on the examples they try, and then it silently fails on a real warehouse schema. The fix isn't a bigger model — it's a small, repeatable eval harness that tells you which model fails where, before it fails in production.

This article walks through building that harness from scratch: a fixed set of schema-to-SQL test cases, a scoring script, and a decision table for interpreting results. It runs on free model access and a free hosted environment, so the whole thing costs nothing but an afternoon.

The actual failure mode

LLM-generated SQL tends to break in predictable places:

1. Join fan-out — the model joins a dimension table that multiplies rows and your totals inflate.
2. Dialect drift — it emits DATE_TRUNC syntax from the wrong warehouse.
3. Hallucinated columns — plausible names that don't exist, especially created_at variants.
4. Aggregation without grouping keys — valid SQL, wrong answer.

A vibe check with two or three prompts will not catch these consistently. A harness with 30–50 fixed cases will.

The artifact: a minimal reproducible harness

The harness has three parts: a golden dataset, a runner, and a scorer. Everything below is designed to be copy-paste runnable. Test it against your own schema — the example uses a tiny synthetic one so there's nothing proprietary in it.

1. Golden dataset (cases.jsonl)

Each line is one test case: a natural-language question, the schema context, and a reference query with its expected result on a seeded fixture database.

{"id": "q01", "question": "Total revenue by month for 2024, ordered by month", "ref_sql": "SELECT date_trunc('month', o.order_date) AS m, SUM(oi.qty * oi.unit_price) FROM orders o JOIN order_items oi ON o.id = oi.order_id WHERE o.order_date >= DATE '2024-01-01' AND o.order_date < DATE '2025-01-01' GROUP BY 1 ORDER BY 1", "tags": ["aggregation", "join"], "trap": "join_fanout"}
{"id": "q02", "question": "Customers who placed more than 3 orders last quarter", "ref_sql": "...", "tags": ["filter", "having"], "trap": "dialect"}

Write 30–50 of these. The trap field is the point: you're documenting the failure you expect models to make, so misses are diagnosable instead of mysterious.

2. Fixture database and runner

Seed a DuckDB or SQLite fixture so expected results are deterministic. DuckDB is nicer here because it speaks a modern SQL dialect and runs in-process.

import duckdb, json

con = duckdb.connect()
con.execute(open("seed.sql").read())  # deterministic fixture data

def run_case(model_fn, case):
    prompt = f"""Schema:\n{open('schema.sql').read()}\n\nQuestion: {case['question']}\nReturn only SQL."""
    candidate = model_fn(prompt)
    try:
        got = con.execute(candidate).fetchall()
        want = con.execute(case["ref_sql"]).fetchall()
        # order-insensitive comparison unless ORDER BY matters
        ok = sorted(map(str, got)) == sorted(map(str, want))
        return {"id": case["id"], "ok": ok, "error": None}
    except Exception as e:
        return {"id": case["id"], "ok": False, "error": str(e)[:200]}

Key detail: compare result sets, not SQL strings. Two queries can look completely different and both be correct — string comparison will drown you in false negatives.

3. Scoring

Aggregate per model, per trap category:

from collections import Counter

def score(results, cases):
    by_trap = Counter()
    for r, c in zip(results, cases):
        if not r["ok"]:
            by_trap[c["trap"]] += 1
    acc = sum(r["ok"] for r in results) / len(results)
    return {"accuracy": round(acc, 3), "misses_by_trap": dict(by_trap)}

Running models for free

The obvious objection: "evals against several models cost money." They don't have to.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access plus a free server option, which covers the two things this harness needs — an LLM endpoint to swap between models, and somewhere to run the runner script on a schedule without keeping a laptop awake. I won't quote specific model names or quotas here because free-tier lineups change; check what's available in your account and plug each endpoint behind the same model_fn interface so swapping is a one-line change:

def make_model_fn(endpoint, model):
    def fn(prompt):
        # your HTTP call here; keep the interface identical across models
        ...
        return sql_text
    return fn

A reasonable workflow on the free server:

1. Nightly cron runs the harness against your current production model — regression detection.
2. When a new model shows up in the free tier, run the same harness once and diff the per-trap misses against your baseline.
3. Only promote a model if it doesn't regress on the trap categories that hurt you historically.

Interpreting results: a decision table

Result pattern	Interpretation	Action
High accuracy, misses concentrated in join_fanout	Model is fine but weak on your schema shape	Add schema hints / few-shot join examples to the prompt
Misses spread evenly across traps	Model is generically weak for this task	Try a different model from the free tier
Dialect errors only	Wrong dialect in prompt context	Specify the engine explicitly; re-run
Parse/exec errors on >10% of cases	Model ignores "SQL only" instruction	Add output parsing, or drop the model

Limitations and honest caveats

• 30–50 cases is a smoke test, not a benchmark. It's enough to rank models for your schema and your question style. It says nothing about general capability, and you should not publish these numbers as model comparisons.
• Free tiers move. Model availability and limits on any free offering can change without notice. Design the harness so the model is a swappable parameter, never a hardcoded dependency.
• Result-set comparison misses semantic issues. A query can return the right rows on a small fixture and still be a performance disaster at scale. If that matters to you, add an EXPLAIN check or a row-count guard on intermediate joins.
• Fixture realism caps value. If your seed data has no edge cases (NULLs, timezone boundaries, duplicate keys), your harness will overestimate every model.

Who should skip this

• If you use LLM-generated SQL a few times a month and always review it by hand, a harness is overkill — keep a checklist instead.
• If your queries touch sensitive schemas you can't put in a prompt to an external endpoint, you need a self-hosted model first; free hosted access is the wrong tool for that constraint.

Wrapping up

The pattern here — fixed cases, deterministic fixtures, result-set comparison, per-trap scoring — transfers to any LLM-assisted task in a data pipeline, not just SQL. The only requirements are a free model endpoint to iterate against and a free place to run the eval on a schedule. If you want to try the workflow, MonkeyCode's free model access and server option are a straightforward way to get both without touching a credit card; the harness above is deliberately provider-agnostic, so bring whatever endpoint you prefer.

Start with ten cases covering the last three SQL bugs that actually bit you. That's usually enough to change which model you trust.