Public text-to-SQL benchmarks use clean schemas with sensible names. Your warehouse has none of those things.
So we benchmarked against the messy kind and published the numbers.
Why academic benchmarks mislead
They measure a real skill — but not the one that determines production outcomes. Enterprise schemas bring:
- Near-duplicate tables where only one is current
- Legacy columns nobody dares drop
- Business rules that exist only in someone's head
- Fiscal calendars that aren't calendar years
- Three plausible join paths where one is right and one is silently wrong
None of that appears in a benchmark built on a tidy schema.
The headline result
| Condition | Accuracy |
|---|---|
| Raw schema access | 14.5% |
| Compiled, governed context | 98.2% |
Same model. Same questions. The only variable was whether the context had been resolved before the model saw it.
That gap isn't model quality. It's the presence or absence of a resolved definition and a proven join path.
What's in the full report
Every major public study collected in one place, plus our own methodology: the query set, the schema characteristics, the scoring rules, and the failure cases — including where we did worse than expected. A benchmark that only shows wins isn't a benchmark, it's marketing with a table in it.
We also report retrieval quality separately: MRR@1 of 0.91 on enterprise concept resolution, because if the correct concept isn't the top result, everything downstream is confidently building on the wrong entity.
How to run your own
Take five questions from your business, at least one unanswerable, and score any vendor on your schema. It takes an afternoon and it's more informative than every benchmark leaderboard combined.
The full benchmark — methodology, query set, every major study compared, and the failure analysis — is here:
👉 The Enterprise Text-to-SQL Accuracy Benchmark: Every Major Study in One Place
Originally published at colrows.com/blogs/text-to-sql-benchmark
Top comments (0)