Building a Zero-Budget Eval Harness for LLM-Assisted SQL Generation

Morgan Li — Wed, 05 Aug 2026 07:06:27 +0000

Most data teams I read about have the same quiet problem: they use an LLM to help write SQL, it works on the examples they try, and then it silently fails on a real warehouse schema. The fix isn't a bigger model — it's a small, repeatable eval harness that tells you which model fails where, before it fails in production.

This article walks through building that harness from scratch: a fixed set of schema-to-SQL test cases, a scoring script, and a decision table for interpreting results. It runs on free model access and a free hosted environment, so the whole thing costs nothing but an afternoon.

The actual failure mode

LLM-generated SQL tends to break in predictable places:

Join fan-out — the model joins a dimension table that multiplies rows and your totals inflate.
Dialect drift — it emits DATE_TRUNC syntax from the wrong warehouse.
Hallucinated columns — plausible names that don't exist, especially created_at variants.
Aggregation without grouping keys — valid SQL, wrong answer.

A vibe check with two or three prompts will not catch these consistently. A harness with 30–50 fixed cases will.

The artifact: a minimal reproducible harness

The harness has three parts: a golden dataset, a runner, and a scorer. Everything below is designed to be copy-paste runnable. Test it against your own schema — the example uses a tiny synthetic one so there's nothing proprietary in it.

1. Golden dataset (`cases.jsonl`)

Each line is one test case: a natural-language question, the schema context, and a reference query with its expected result on a seeded fixture database.

{"id": "q01", "question": "Total revenue by month for 2024, ordered by month", "ref_sql": "SELECT date_trunc('month', o.order_date) AS m, SUM(oi.qty * oi.unit_price) FROM orders o JOIN order_items oi ON o.id = oi.order_id WHERE o.order_date >= DATE '2024-01-01' AND o.order_date < DATE '2025-01-01' GROUP BY 1 ORDER BY 1", "tags": ["aggregation", "join"], "trap": "join_fanout"}
{"id": "q02", "question": "Customers who placed more than 3 orders last quarter", "ref_sql": "...", "tags": ["filter", "having"], "trap": "dialect"}

Write 30–50 of these. The trap field is the point: you're documenting the failure you expect models to make, so misses are diagnosable instead of mysterious.

2. Fixture database and runner

Seed a DuckDB or SQLite fixture so expected results are deterministic. DuckDB is nicer here because it speaks a modern SQL dialect and runs in-process.

import duckdb, json

con = duckdb.connect()
con.execute(open("seed.sql").read())  # deterministic fixture data

def run_case(model_fn, case):
    prompt = f"""Schema:\n{open('schema.sql').read()}\n\nQuestion: {case['question']}\nReturn only SQL."""
    candidate = model_fn(prompt)
    try:
        got = con.execute(candidate).fetchall()
        want = con.execute(case["ref_sql"]).fetchall()
        # order-insensitive comparison unless ORDER BY matters
        ok = sorted(map(str, got)) == sorted(map(str, want))
        return {"id": case["id"], "ok": ok, "error": None}
    except Exception as e:
        return {"id": case["id"], "ok": False, "error": str(e)[:200]}

Key detail: compare result sets, not SQL strings. Two queries can look completely different and both be correct — string comparison will drown you in false negatives.

3. Scoring

Aggregate per model, per trap category:

from collections import Counter

def score(results, cases):
    by_trap = Counter()
    for r, c in zip(results, cases):
        if not r["ok"]:
            by_trap[c["trap"]] += 1
    acc = sum(r["ok"] for r in results) / len(results)
    return {"accuracy": round(acc, 3), "misses_by_trap": dict(by_trap)}

Running models for free

The obvious objection: "evals against several models cost money." They don't have to.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode currently offers free model access plus a free server option, which covers the two things this harness needs — an LLM endpoint to swap between models, and somewhere to run the runner script on a schedule without keeping a laptop awake. I won't quote specific model names or quotas here because free-tier lineups change; check what's available in your account and plug each endpoint behind the same model_fn interface so swapping is a one-line change:

def make_model_fn(endpoint, model):
    def fn(prompt):
        # your HTTP call here; keep the interface identical across models
        ...
        return sql_text
    return fn

A reasonable workflow on the free server:

Nightly cron runs the harness against your current production model — regression detection.
When a new model shows up in the free tier, run the same harness once and diff the per-trap misses against your baseline.
Only promote a model if it doesn't regress on the trap categories that hurt you historically.

Interpreting results: a decision table

Result pattern	Interpretation	Action
High accuracy, misses concentrated in `join_fanout`	Model is fine but weak on your schema shape	Add schema hints / few-shot join examples to the prompt
Misses spread evenly across traps	Model is generically weak for this task	Try a different model from the free tier
Dialect errors only	Wrong dialect in prompt context	Specify the engine explicitly; re-run
Parse/exec errors on >10% of cases	Model ignores "SQL only" instruction	Add output parsing, or drop the model

Limitations and honest caveats

30–50 cases is a smoke test, not a benchmark. It's enough to rank models for your schema and your question style. It says nothing about general capability, and you should not publish these numbers as model comparisons.
Free tiers move. Model availability and limits on any free offering can change without notice. Design the harness so the model is a swappable parameter, never a hardcoded dependency.
Result-set comparison misses semantic issues. A query can return the right rows on a small fixture and still be a performance disaster at scale. If that matters to you, add an EXPLAIN check or a row-count guard on intermediate joins.
Fixture realism caps value. If your seed data has no edge cases (NULLs, timezone boundaries, duplicate keys), your harness will overestimate every model.

Who should skip this

If you use LLM-generated SQL a few times a month and always review it by hand, a harness is overkill — keep a checklist instead.
If your queries touch sensitive schemas you can't put in a prompt to an external endpoint, you need a self-hosted model first; free hosted access is the wrong tool for that constraint.

Wrapping up

The pattern here — fixed cases, deterministic fixtures, result-set comparison, per-trap scoring — transfers to any LLM-assisted task in a data pipeline, not just SQL. The only requirements are a free model endpoint to iterate against and a free place to run the eval on a schedule. If you want to try the workflow, MonkeyCode's free model access and server option are a straightforward way to get both without touching a credit card; the harness above is deliberately provider-agnostic, so bring whatever endpoint you prefer.

Start with ten cases covering the last three SQL bugs that actually bit you. That's usually enough to change which model you trust.

DEV Community: Morgan Li