How to Actually Measure Whether Your Text-to-SQL Is Any Good

You wired an LLM up to your database. You asked it "how many active users signed up last month?", it wrote a tidy SELECT, the number looked plausible, and everyone in the demo nodded. Ship it.

Then a founder asks the same question a slightly different way, gets a number that's off by 20%, and now nobody trusts the feature. The uncomfortable truth about text-to-SQL is that a query that runs tells you almost nothing about whether it's right. "No error" and "correct answer" are two completely different things, and the gap between them is where trust quietly dies.

If you're building a natural-language query feature — an internal analytics box, a customer-facing "ask your data" panel, an AI assistant hooked to production — you need a way to measure accuracy that goes beyond "it looked fine when I tried it." This post walks through how text-to-SQL is actually evaluated, the sharp edges in those metrics, what the famous benchmarks do and don't prove, and how to build a lightweight eval suite for your own schema.

"It ran" is the weakest possible signal

Consider a question against a typical SaaS schema:

"How much revenue did we make from paid plans in July?"

Here are two queries an AI might produce:

-- Query A
SELECT SUM(amount)
FROM invoices
WHERE status = 'paid'
  AND created_at >= '2026-07-01'
  AND created_at <  '2026-08-01';

-- Query B
SELECT SUM(amount)
FROM invoices
WHERE created_at >= '2026-07-01'
  AND created_at <  '2026-08-01';

Both run. Both return a single number. On a test database where every July invoice happens to be paid, both return the same number. Query B is wrong — it silently includes refunded and failed invoices — but nothing about the execution surfaces that. This is the core problem: to evaluate text-to-SQL you have to compare against a notion of correct, not just runnable.

The three ways people measure accuracy

There are three broad approaches, and they trade off strictness against fairness.

Metric	How it works	Weakness
Exact-set match (ESM)	Compares the SQL text/clauses against a reference query, component by component	Punishes correct queries that are written differently
Execution accuracy (EX)	Runs both queries and compares the result sets	Two different queries can return the same rows by accident
Semantic equivalence	Judges whether two queries mean the same thing (often via an LLM or query analysis)	Harder to automate, can itself be wrong

Exact-set match is the oldest and the most brittle. Reference SQL says WHERE status = 'paid'; the model writes WHERE status IN ('paid'). Identical meaning, "wrong" by string comparison. ESM's rigid matching overlooks semantically correct but stylistically different queries, so it systematically under-counts good answers.

Execution accuracy fixed the obvious flaw: instead of comparing text, run both queries and compare the results. It has become the dominant metric in modern benchmarks precisely because it treats syntactically distinct but semantically equivalent queries as equal. If two queries produce the same rows, who cares how they're written?

The catch is that execution equality does not imply semantic equivalence — which brings us to the trap.

The trap: execution accuracy lies on small test data

Go back to Query A and Query B above. On a sparse test database, they agree. That's a false positive: a wrong query scored as correct because the test data wasn't diverse enough to expose the difference. An incomplete WHERE clause slips through whenever every row in the test set happens to satisfy the missing condition.

This isn't a rare edge case. Studies of execution-based evaluation have measured false-positive rates around 11% — roughly one in nine "correct" queries is actually wrong and just got lucky on the test data. If your eval database is a handful of tidy demo rows, that number is worse, not better.

The fix that the research community landed on is elegant: test-suite evaluation. Instead of one small database, you evaluate the query against several databases specifically constructed so that a wrong query is very likely to diverge from the right one on at least one of them. The idea (from Semantic Evaluation for Text-to-SQL with Distilled Test Suites) is to distill a compact set of databases that achieves high code coverage of the reference query, giving a tight approximation of true semantic accuracy without needing to prove equivalence formally.

You can apply the spirit of this cheaply. Seed your test data with rows that would break a lazy query:

-- Adversarial seed rows for the revenue example
INSERT INTO invoices (amount, status, created_at) VALUES
  (100, 'paid',     '2026-07-15'),  -- should count
  (100, 'refunded', '2026-07-16'),  -- must NOT count
  (100, 'failed',   '2026-07-17');  -- must NOT count

Now Query A returns 100 and Query B returns 300. The bug is visible. A good eval dataset is one where being sloppy costs you.

What Spider and BIRD actually tell you

Two public benchmarks dominate the conversation, and it's worth knowing what each is really measuring before you quote a leaderboard number to your team.

Spider contains 10,181 questions over 5,693 unique queries across 200 databases spanning 138 domains. Its whole point is cross-domain generalization: the test databases are unseen at training time, so a high score means the model can handle a schema it has never met. That maps well to "will this work on my customers' databases," which are all different.

BIRD is the more real-world sibling — over 12,000 queries across 95 databases in 37 professional domains — and it adds two things Spider mostly ignores. First, it rewards using external knowledge (the messy business context real questions require). Second, it measures efficiency, not just correctness, via a Valid Efficiency Score: a query that returns the right rows but does a full table scan where an indexed lookup would do scores lower than a fast, correct one.

That efficiency dimension matters more than people expect. Two queries can both be "correct" and differ wildly in cost:

-- Correct but expensive: function on the column kills the index
SELECT * FROM events
WHERE DATE(created_at) = '2026-07-15';

-- Correct and cheap: sargable range keeps the index usable
SELECT * FROM events
WHERE created_at >= '2026-07-15'
  AND created_at <  '2026-07-16';

Both return the same rows. In a customer-facing feature, only one of them is acceptable at scale.

The thing to remember: a leaderboard score is measured on someone else's schemas and questions. It's a useful signal for picking a model, but it is not a substitute for testing against your database and your users' phrasing.

Building your own eval suite

The good news is you don't need a research pipeline. A useful eval is just a set of question → reference-query pairs (call them golden queries) plus a harness that runs the model's output and compares result sets.

Start by collecting real questions — from support tickets, from analysts' saved queries, from whatever people actually ask. For each, write the SQL you know is correct:

-- golden_queries.yaml (conceptually)
-- id: revenue_paid_july
--   question: "How much revenue from paid plans in July?"
--   sql: |
--     SELECT SUM(amount) FROM invoices
--     WHERE status = 'paid'
--       AND created_at >= '2026-07-01'
--       AND created_at < '2026-08-01';

Then the harness, in pseudocode:

passed = 0
for case in golden_queries:
    predicted_sql = model.generate(case.question, schema)
    try:
        got = db.run(predicted_sql)          # your candidate
    except SQLError:
        record(case, "invalid_sql"); continue
    want = db.run(case.sql)                   # your golden answer
    if result_sets_equal(got, want):         # order-insensitive compare
        passed += 1
    else:
        record(case, "wrong_result", got, want)

print(f"Execution accuracy: {passed / len(golden_queries):.1%}")

A few details make this far more honest than a naive version. Compare result sets, not row order, unless the question asked for a specific ordering — otherwise a correct GROUP BY fails just because rows came back shuffled. Run every case against the adversarial seed data from earlier so false positives can't hide. And bucket your failures — invalid SQL, wrong table, missing filter, wrong aggregation — because "62% accurate" is far less useful than "most failures are a missing tenant filter."

If you'd rather not run the model against production directly during all this, a read-only gateway helps: managed MCP servers like Draxlr expose a database to an AI client over a read-only (SELECT-only) connection, which is a sane place to point an eval harness so a buggy generated query can't do anything but return wrong rows.

Common mistakes and gotchas

The most common one is treating a green demo as evidence. A feature that answers five questions correctly on stage can be 60% accurate across the long tail of real phrasings, and you'll never know until you measure the tail.

The second is a tiny, tidy eval database. Sparse data is exactly what manufactures false positives; if every row satisfies the filters your model forgets, your eval will happily bless broken SQL. Diverse, adversarial rows are a feature.

The third is scoring only correctness and ignoring cost. A query that's right but scans a 200-million-row table is a production incident waiting to happen — track query time or plan cost alongside accuracy.

Fourth: overfitting to the questions you happened to write. If your golden set is 30 questions and you tune prompts until all 30 pass, you've built a model that's great at those 30 questions — keep a held-out set you don't tune against. And version the eval itself: when you change the prompt, schema description, or model, the number that matters is the delta from a saved baseline, not the absolute score.

Key takeaways

Evaluating text-to-SQL is about measuring correctness, not runnability. Exact-match under-counts good queries; execution accuracy is the practical standard but produces false positives on thin test data; test-suite-style adversarial data is how you close that gap. Public benchmarks like Spider (cross-domain generalization) and BIRD (real-world knowledge plus efficiency) are great for choosing a model but never a substitute for testing on your own schema. Build a golden-query set from real questions, run it against deliberately diverse data, compare result sets rather than SQL text, track cost as well as correctness, and re-run it on every change. That's the difference between "it worked in the demo" and "we know it's 94% accurate and we watch that number."

How are you measuring your text-to-SQL feature today — golden queries, LLM-as-judge, eyeballing, or nothing yet? Drop your setup (and your favorite false-positive horror story) in the comments.