Half the answer keys in text-to-SQL benchmarks are wrong. So I generated the database from the answer key.

Earlier this year a team at UIUC audited the benchmarks the text-to-SQL field ranks itself on. The numbers in the paper (Jin et al., VLDB 2026) stopped me cold: 52.8% of BIRD Mini-Dev annotations and 62.8% of Spider 2.0-Snow annotations are wrong. Not ambiguous. Wrong, as adjudicated by human SQL experts after an agent flagged them. When the authors corrected just 100 examples and re-evaluated 16 open-source agents, relative performance swung by as much as 31% and leaderboard positions moved by up to 9 places.

Every team that picked a data agent off those leaderboards was, to a measurable degree, choosing based on noise.

I build synthetic data tooling as a side quest, and my whole library is organised around one guarantee: declared numbers hold exactly in generated data. Reading that paper, it was hard to miss that the two problems are mirror images. Benchmarks struggle to attach correct answers to existing databases. My engine attaches a database to declared answers. So I spent this week building the inversion, and this post walks through the mechanism, the verification, and the parts that did not survive their own verification, because those are the most instructive.

Why annotation-after-the-fact fails structurally

The error rates are not annotator laziness. Look at how every benchmark in this lineage gets built. A database exists first, scraped or donated or assembled. Then question writers read schemas they did not design and write natural-language questions. Then annotators produce the gold SQL and expected answers by interrogating data full of null conventions, encoding quirks, and domain definitions they can only infer.

The UIUC audit's taxonomy shows exactly where that leaks. Semantic mismatches between question and SQL, like a strict inequality in the question compiled as an inclusive BETWEEN. Schema misunderstandings: missed aggregations, wrong joins, a filter on the wrong status column. Domain knowledge errors, their example being an annotator treating K-12 as equivalent to grades 1 through 12. And genuinely ambiguous questions with several defensible readings.

Strip away the details and all of these share one structural property: they can only exist because the answer was derived after the data, by a human reading it. The answer key is a claim about the database, and claims can be wrong.

Inverting the construction

So make the answer key not a claim but a constraint.

Declare the ground truth first, as a specification: exact monthly revenue for calendar 2025, a fraud rate rising from 2% to 3.5%, two foreign-key relationships that must have zero orphans. Then generate a database that satisfies the specification. Questions are derived from the declaration. The expected answer for "total order amount in November 2025" is not something anyone computed by reading rows. It is $750,000.00 because the data was constructed so that it would be.

The generation math matters here, so briefly: for each declared period the engine allocates a row count from the target and the average transaction value, draws per-row amounts from a lognormal around that average, then rescales the period so it sums to the target exactly. Two stages, allocation then scaling, no learned model anywhere in the path, fully deterministic given a seed. The method is written up in the Misata paper; the engine is open source. The property that makes it usable for evaluation is that the aggregate guarantee survives all the way to files on disk: modern float serialization round-trips exactly, so the CSV a grader reads carries the same cent-exact sums the generator produced.

Generators don't get to grade their own homework

Here is the part I consider non-negotiable, because "the generator says the generator is right" should convince nobody.

After the CSVs are written, every candidate question's gold SQL is executed by DuckDB against those files, the ones that actually ship, and the observed answer is compared with the declared one. DuckDB shares no code with my engine. The two systems agree on nothing except the files. Each shipped question looks like this, from the actual pack:

{
  "id": "q011",
  "question": "In the orders table, what is the total of amount during November 2025 (rows whose timestamp is on or after 2025-11-01 and strictly before 2025-12-01)? Give a number rounded to 2 decimal places.",
  "gold_sql": "SELECT ROUND(SUM(\"amount\"), 2) FROM \"orders\" WHERE \"order_date\" >= TIMESTAMP '2025-11-01 00:00:00' AND \"order_date\" < TIMESTAMP '2025-12-01 00:00:00'",
  "expected_answer": 750000.0,
  "answer_type": "number",
  "round_decimals": 2
}

Notice the question states its window as half-open and names its rounding. That is deliberate. Most of what the audit classified as ambiguity comes from questions that leave boundaries and output format to the reader's imagination. When you author the question from the declaration instead of from the data, you can simply say what you mean.

The demonstration pack is public: github.com/rasinmuhammed/evalpacks. A 73,629-row orders table plus customers and products, 35 shipped questions, and a certificate recording the DuckDB version, the seed, the spec hash, and every per-question verification result. Re-checking it needs one dependency and about thirty seconds:

pip install duckdb
python pack/verify.py
# 35/35 verified exactly

Edit one amount in one CSV and the script exits nonzero. That is the entire trust model. There is no leaderboard to take on faith, just a check anyone can run against the artifact itself.

The questions that did not survive

The pack shipped 35 questions. The spec produced 40 candidates. The other five are my favorite part of the artifact.

Declared rates collide with integer arithmetic. January was declared at a 2% fraud rate, and the January allocation came out to 4,824 orders. Two percent of 4,824 is 96.48 orders, and you cannot flag half an order. The achievable neighbors are 96 flagged rows, which is 1.99%, or 97, which is 2.01%. The verification gate measured 1.99% against a declared 2.00% and refused to ship the question. Same story for four other months:

period	declared rate	achieved rate	shipped?
2025-01	2.00%	1.99%	dropped
2025-06	2.68%	2.69%	dropped
2025-08	2.95%	2.96%	dropped
2025-10	3.23%	3.22%	dropped
2025-11	3.36%	3.37%	dropped

Seven other monthly rate anchors happened to be exactly achievable at four decimal places, and those shipped. All five failures are recorded in the manifest with their observed values, next to the full generation spec.

I want to be precise about what this section is, because a reader could mistake it for an admission. A benchmark that publishes what it refused to include, with the measured reason, is making a stronger claim than one that ships everything: it is telling you the shipping bar is exact verification and showing you the bar rejecting things. The alternative, quietly shipping a question whose true answer is 1.99% with an answer key that says 2%, is precisely the failure mode the UIUC paper documented at scale.

Regeneration, or why contamination stops mattering

BIRD and Spider are in every frontier model's training corpus by now. That is not an accusation, it is arithmetic: benchmark questions get indexed, discussed, and absorbed. Static benchmarks depreciate.

An evalpack is a spec plus a seed, so it regenerates. Change the seed and you get a different database that satisfies the same declared answers. From my own run while writing this post:

seed 20260710: November total = 750,000.00 over 73,629 orders
seed 99:       November total = 750,000.00 over 73,629 orders
shared order_ids between the two databases: 544

Two databases, essentially disjoint rows (the 544 shared IDs are random collisions in a ten-million-value ID space), identical declared answers, and the seed-99 pack re-verifies 35/35 on its own certificate. If you suspect a model has seen your eval data, you rotate the seed and the answer key survives. No annotation cycle, no re-adjudication.

There is a second contamination defense hiding in plain sight: every entity in the data is plausible fiction. No model can answer "which month had the highest revenue" from memorized world knowledge about a retailer that does not exist. It has to query. For evaluation data, fictional values are not a compromise. They are the point.

Rewards for RL, where wrong labels hurt twice

If you train data agents with reinforcement learning, ground-truth noise is worse than in evaluation, because the policy optimizes toward the noise. The large synthetic SQL corpora used for training measure around 86% full correctness by their own published human evals, and RL on a 14%-wrong reward signal is a strange thing to spend GPUs on.

A verified evalpack is, from an RL perspective, a reward function with a certificate: exact-match rewards whose correctness was checked by an independent engine before training started, generable at whatever scale and difficulty the spec dials in. Join depth, aggregation complexity, and temporal structure are parameters, not accidents of whichever database was available. I think this is where answer-key-first construction ends up mattering most, and it is the part I would most like to compare notes on with people building agent environments.

Design choices a skeptic should ask about

Three decisions define the current scope, and each was made with eyes open.

Question families cover declared aggregates, per-period counts, rates, extrema, and referential integrity, not arbitrary SQL. That is what the spec language can declare today. The right way to extend coverage is to extend the declaration language, keeping the property that every shipped answer traces to a declared constraint, rather than annotating free-form questions and reintroducing the original disease.

Natural-language ambiguity is reduced, not eliminated. Authoring questions from the declaration lets every question state its window, its rounding, and its tie-breaking rule explicitly, which removes the largest documented ambiguity sources. Wording is still wording, and I will not claim a theorem where I only have a discipline.

And this complements human-authored benchmarks rather than replacing them. Real-world schema messiness, dialect coverage, the long tail of enterprise weirdness: existing benchmarks carry that, and correcting them, as the UIUC team is doing, stays essential. Answer-key-first construction owns the complementary territory: the error classes where annotation-after-the-fact structurally fails, at whatever scale you need, contamination-free by rotation.

Try it

Everything is MIT licensed. The pack, certificate, and verifier are at github.com/rasinmuhammed/evalpacks. Building your own takes one install and one command:

pip install 'misata[evalpack]'
misata evalpack --config your_schema.yaml -o your_pack --seed 42

The declaration language, integrity proofs, and the rest of the engine are documented at misata.studio/docs, and there is a no-code version at misata.studio if you want to go from a plain-English description to a generated database without writing the schema by hand.

If you build evaluations for data agents, or RL environments that contain databases, and this construction is either useful to you or broken in a way I have not noticed, I genuinely want to hear it. Open an issue on the repo. The fastest way to make the answer key trustworthy is to keep trying to break it.