I've been experimenting with pointing LLMs at databases through MCP servers and asking them questions. The results followed a pattern. First, the model would simply make up tables and columns that didn't exist, then write plausible-looking SQL against them.
Giving it tools to investigate the schema fixed the first failure: it could list tables and describe columns, and it stopped making up structure it could now look up. But a subtler failure was left behind. When it wasn't clear what a column meant, the model didn't ask and didn't flag it. It picked an interpretation and ran with it.
That second failure is the one worth writing about. The models are decent at this point, and I don't think a better one is what's missing. What's missing is on our side: the context that tells the model what the data actually means.
The Benchmark Cliff
For years, text-to-SQL progress was measured on Spider, an academic benchmark with clean schemas, a handful of tables per database, and readable column names. The top system passed 91% execution accuracy and the leaderboard was frozen in early 2024, effectively solved.
Then the same lab released Spider 2.0, built from real enterprise warehouses instead: BigQuery and Snowflake databases that often exceed 1,000 columns, where answers can take over 100 lines of SQL. GPT-4o scored 86.6% on academic Spider and 10.1% on Spider 2.0, the same model against real schemas.
BIRD, another major benchmark, sampled 500 of ChatGPT's failures and categorized them: 41.6% linked to the wrong tables or columns, and another 40.8% misunderstood the database contents or generated schema items that don't exist. Over 80% of failures are schema grounding. SQL syntax barely registers as a category. The failures aren't in the SQL. They're in the model's picture of what the database means.
The Warehouse You'd Actually Point It At
Ask two teams at the same company for last quarter's revenue and you can get two different numbers, both correct by their own definition. One excludes returns, one doesn't. One counts bookings, one counts recognized revenue. The metric name is shared; the logic behind it is not. That logic usually isn't in the warehouse at all, it's in a Slack thread from 2023 and the head of the analyst who built the dashboard.
The physical layer is no better. A warehouse that has been through a few years of team changes looks something like this:
SHOW TABLES LIKE 'customers%';
-- customers
-- customers_v2
-- customers_final
The migration behind customers_v2 got most of the way done before priorities shifted. Columns are the same story: amt and amt_2 sit next to each other, no comment on either. A model with schema access can see all of this. It cannot see which table is canonical, because nobody wrote that down anywhere a query can reach.
A Hacker News commenter describing their experience with Databricks Genie summed up the trap: "I have hundreds of tables designed by several different teams. I do have decent documentation on the tables but if I had a nice, organized data model I wouldn't need an AI assistant." Text-to-SQL works best on exactly the warehouses that need it least.
Same Model, Different Context
The evidence I trust most comes from cases where the model stays fixed and only the context changes.
Pinterest built text-to-SQL internally and found the bottleneck wasn't SQL generation, it was finding the right table among hundreds of thousands. Adding table documentation to their search index moved the right-table hit rate from about 40% to 90%, with the model unchanged.
BIRD ships each question with an optional sentence of human-written context about the data. That one sentence lifts GPT-4 by about 20 points. The part I find most convincing: it lifts human accuracy by the same 20 points, from 72% to 93%. When missing context hurts experts as much as it hurts the model, the problem isn't intelligence. The information just isn't there.
Even the vendor numbers say this. Snowflake's engineering blog reports raw GPT-4o at 51% on their real-world evaluation, and 90%+ once a human hand-builds a semantic model describing the data. They're selling Cortex Analyst, so it's a vendor benchmark. But what it concedes is the point: the model alone is a coin flip, and the entire difference is metadata someone wrote.
The Agents Got Good, Though
The counterargument has real numbers behind it. Spider 2.0 scores went from around 10% in late 2024 to claimed 90%+ on its Snowflake track by mid-2026, and dbt's benchmark measured raw text-to-SQL accuracy nearly doubling between 2023 and 2026 from model progress alone, 32.7% to 64.5%. If the warehouse were the whole problem, that shouldn't have happened.
But the gains happened in a specific way. The systems topping Spider 2.0 aren't single prompts against a raw schema. They're agents that profile the database, read the metadata and dialect docs, sample values, and iterate against execution errors. They climbed the leaderboard by doing metadata work at query time. That's not evidence against the thesis, it's the same fix applied expensively, on every query, by the model.
And there's a limit to what probing can recover. An agent can discover that customers_v2 exists, what types its columns hold, which values amt_2 takes. It can't discover which of the three revenue definitions finance considers correct, because that fact isn't in the database. It isn't anywhere the agent can reach.
That was my MCP experience in miniature. Investigation tools stopped the hallucinations. Nothing the tools could reach told the model what the data meant.
What Actually Works
Read the vendors' own documentation and the answer is already there. Databricks Genie's best-practices page recommends starting with five or fewer tables and a curator who understands the data. Snowflake's Cortex Analyst has you author a semantic model, because schemas on their own lack the business definitions that queries depend on. Amazon Q wants curated topics and human-reviewed verified answers. Each of these is sold as an AI product, and each ships with the same homework attached: document your data first.
The dbt benchmark put numbers on the homework. Adding three dbt models of light data modeling pushed raw text-to-SQL from the mid-60s to 84-90%. Going through their semantic layer reached 98-100% on in-scope questions, and the failures changed shape, from a plausible wrong answer to an error message. That last part matters more than the accuracy. A refusal is annoying but visible. A plausible wrong number can make it into a board deck before anyone checks.
There's a pattern in who makes this argument in public. Collate sells a metadata platform. Omni and dbt sell semantic layers. a16z has portfolio companies to promote. That doesn't make them wrong, I've been citing their numbers all post. But it explains why "document your warehouse" keeps arriving wrapped in a product. Strip the packaging and what's left is the advice data teams have been ignoring for a decade: model the data, write down the definitions, mark what's canonical. The semantic layer is documentation debt paydown with a better name.
Text-to-SQL exposed a problem that was already there. Undocumented warehouses worked because analysts carried the missing context in their heads and quietly compensated on every query. Point a model at the same warehouse and there's no one left to compensate. If you want it to work, the job is the same one it's always been: model the data, write down what things mean, mark what's canonical. That job was just easy to postpone when a human was absorbing the cost.
Top comments (0)