Text-to-SQL Accuracy Isn't a Model Problem. It's a Schema Problem.

TLDR: dbt Labs ran the same eleven questions four different ways and found that modelling the schema — with no semantic layer involved at all — moved text-to-SQL accuracy from 64.5% to 90.0%. Adding a semantic layer on top of that modelled schema moved it a further 8 points. Snowflake, using a semantic model across four BIRD databases, measured a 21-point average lift. MotherDuck pointed three frontier models at small, clean schemas with nothing but the DDL and reported 95% — though those same runs score 58–64% under BIRD's strict scoring. Read side by side, these look contradictory. They aren't. None of them is measuring the model. They're measuring how much translation work somebody already did to the schema, and how forgivingly the answers were graded.

Ask an AI assistant to write a query against your warehouse and it will, almost every time, produce something that runs. That's the trap. A query that executes and returns a number looks exactly like a query that answers the question — right up until someone downstream builds a forecast on it.

Two queries, one question, two answers

Here's the shape this takes on an ordinary warehouse, away from the benchmark leaderboards. A SaaS company wants last month's revenue. An AI assistant, asked two reasonable questions, produces two queries.

-- Query A: "what was our revenue last month?"
SELECT SUM(amount_due) AS revenue
FROM invoices
WHERE status = 'paid'
  AND billing_period = '2026-08';

-- Query B: "break down last month's revenue by product"
SELECT p.name AS product,
       SUM(i.amount_due) AS revenue
FROM invoices i
JOIN invoice_line_items ili ON ili.invoice_id = i.invoice_id
JOIN products p ON p.product_id = ili.product_id
WHERE i.status = 'paid'
  AND i.billing_period = '2026-08'
GROUP BY p.name;

-- Query A result:
revenue = 1,148,000

-- Query B result:
product            revenue
-----------------  ---------
Platform (base)    1,664,000
Extra seats          988,000
Usage overage        705,000
-----------------  ---------
total              3,357,000

Both queries ran without error. Both came from reasonable, good-faith prompts. But Query B sums i.amount_due — the invoice total — once for every line item attached to that invoice. A customer with a base plan, an extra seat and a usage overage contributes their entire invoice total three times, once under each product. "Platform (base)" doesn't show platform revenue; it shows the full value of every invoice that happened to contain a platform line.

The fix is one column: sum ili.line_amount, the line's own amount, not the invoice's. Nobody wrote a bug. The join changed what one row means, and SUM kept adding as if it hadn't. If Query B's breakdown reaches a board deck before anyone reconciles it against Query A, that takes an awkward meeting to walk back.

This is not a model failure. Point a frontier model at this schema and it will make the same mistake confidently and articulately, because nothing in the schema says that invoice_line_items has a different grain than invoices, or that amount_due is only additive at the invoice level. That knowledge lives in someone's head, or it lives in a semantic layer. If it lives in neither, which query you get is a coin flip.

The spread that looks like disagreement

Three of the most-cited text-to-SQL results around, read together, look like they're arguing. Read separately, each is internally consistent — which is the tell that they're measuring different things.

• dbt Labs' April 2026 benchmark tested four configurations over an insurance dataset, eleven questions, twenty runs per model. The one people quote is text-to-SQL versus semantic layer on a modelled project: Claude Sonnet 4.6 at 90.0% against 98.2%, GPT-5.3-Codex at 84.1% against 100.0%. The more interesting comparison is the one underneath it. Against the original normalised tables, plain text-to-SQL managed 64.5% across all eleven questions; the same method against a modelled project hit 90.0%. That 25.5-point improvement came from modelling alone, with no semantic layer anywhere in the picture — and dbt built it by prompting an LLM to write "as few dbt models as possible," which produced just three. Their summary: "Adding even minimal modeling on top of raw tables improved results across the board."
• Snowflake's semantic-model evaluation (March 2025, using Claude 3.5 Sonnet) is often cited as "57% to 78% on BIRD." It's really an average across four BIRD databases, and the spread inside it matters: debit_card_specializing went 52% → 83%, california_schools 63% → 80%, thrombosis_prediction 45% → 70%, toxicology 69% → 79%. Averaged, an "approximately 20% increase in accuracy" — 21 points — from adding a semantic model, not from a newer model.
• MotherDuck ran the opposite experiment, pointing Claude Opus 4.5, GPT-5.2 and Gemini 3 Flash at 500 BIRD questions with, in their words, "No semantic layer. No query history. No special context. Just the schema." The reported result is 95%. That figure sits at the top of a four-tier evaluation ladder, and the bottom rung is a lot less flattering: 64.0% (train) and 58.2% (test) under strict BIRD execution matching, rising through correction of known benchmark errors and tolerance for formatting differences to 94.9%/94.4% once an LLM judges whether the answer is defensible. Their argument is two-part — that BIRD's strict scoring is itself misleading, because "to score above 62% under strict rules, you have to start reproducing the benchmark's mistakes," and that "Good data modeling is the semantic layer."

So there are two variables moving here, not one, and honest reading requires separating them. Scoring strictness explains most of the distance between MotherDuck's 95% and everyone else's numbers; you cannot line that figure up against dbt's or Snowflake's and treat the gap as real capability. Schema modelling explains the rest, and it's the variable you control. dbt moved text-to-SQL a long way with three models and no semantic layer. Snowflake moved it 21 points with a semantic model over databases they didn't restructure. MotherDuck got high marks with no layer at all — over BIRD databases that, as they note, average seven tables. None of these is a claim about which model writes better SQL.

There's a ceiling worth keeping in view too. BIRD's own paper reports that its human baseline — data engineers and database students — reached "the human result of 92.96%" execution accuracy. People who write SQL for a living, working on databases of seven tables, get roughly one question in fourteen wrong. The bar was never "flawless." It's "at least as reliable as the analyst who used to own this."

What "the schema needs modelling" actually means

"Modelling" sounds like an abstraction exercise. In practice, on a schema like the one above, it's answering three concrete questions before anything touches a model:

What is the grain of each table, in one sentence? invoices is one row per invoice. invoice_line_items is one row per line on an invoice. Join them and your result set's grain becomes "one row per line item" — so any SUM of an invoice-level column is now double-, triple- or n-counting, depending on how many lines the average invoice carries. That single fact is what would have caught Query B.

Which columns carry business meaning that isn't in their name? status = 'paid' looks unambiguous. Does it need to exclude refunds? Is a partially refunded invoice still "paid" for revenue purposes? These are policy decisions someone made once, verbally, in a meeting nobody minuted — and no assistant can recover them from the schema alone.

Is there more than one path to the same number? Revenue might come from invoices.amount_due, from summing invoice_line_items.line_amount, or from an mrr_snapshots table a finance job populates nightly. Three legitimate paths, three numbers that won't quite agree, and none of them "wrong" — they answer slightly different questions that all get asked as "what's our revenue."

"Just model your warehouse" is easy advice to give

The obvious objection to all of this: dbt's benchmark fixed its schema with three new models over an eleven-question insurance dataset. You have four hundred tables, three teams who each define revenue differently, and a migration budget of zero. "Model the schema properly" is not a thing you can do this quarter, and anyone who has tried knows that the modelling is the easy half — getting three departments to agree which definition wins is the hard one.

That constraint is exactly why semantic layers exist. They're the retrofit path: a place to encode grain, join paths and metric definitions without restructuring the warehouse underneath. Snowflake's 21-point average came this way, over databases nobody rebuilt. Treating "clean DDL" and "semantic layer" as interchangeable is true for benchmarks and false for anyone with legacy tables and a roadmap.

There's also a second argument for the retrofit that the accuracy columns actively hide. In dbt's unmodelled configuration, the semantic layer scored 0.0% on the subset of questions requiring too many joins to resolve — not because it answered them wrongly, but because it declined to answer at all. As dbt puts it: "the Semantic Layer tells you it can't answer. It never returns invalid data. Text-to-SQL will cheerfully give you a wrong number." Plain text-to-SQL scored 70–100% on that same subset, which looks like a win until you remember that nothing in those results distinguishes a correct answer from a confident one. A coverage gap you can see beats a coverage gap that quietly returns 3,357,000. That asymmetry appears in no accuracy column anywhere, and for a number headed to a board deck it may matter more than the points do.

What both routes share is scope. Neither dbt's three models nor a semantic layer requires modelling the whole warehouse — only the slice that answers the questions people actually ask. That's the practical move: take the handful of questions your team asks weekly, model just the tables those touch, and leave the other three hundred and eighty alone until someone asks them something.

The benchmarks didn't find a model problem. They found a translation-work problem, and measured what happens when someone does the translation once instead of leaving it to a fresh guess on every query.

The checklist that transfers to your warehouse

None of the studies above tells you which side of the gap your own schema sits on. Run this instead:

1. Take your five most-asked business questions and write down, for each table involved, its grain in one sentence. If you can't do that quickly, an assistant generating SQL against those tables can't either.
2. Find every column whose name promises more precision than its values deliver — status, type, category especially. If status needs a footnote to interpret correctly, that footnote has to live somewhere the assistant can read: a view, a semantic layer, a column comment. Not a Slack thread from 2024.
3. Ask the same business question two ways and see whether you get two numbers. That's what happened above. If it happens on your warehouse, you have a modelling gap rather than an AI-competence gap, and no model upgrade closes it.

The numbers make the same point from three directions: 90% accuracy sounds like a solved problem until you notice the missing 10% is where double-counted joins and silently redefined metrics live, invisible precisely because the query still runs. Building the judgment to catch that — reading a join and knowing what it did to the grain, reconciling a number against a second path before trusting it — is what SophiArch's SQL for Data Analysis course is built around, with a dedicated lesson on joins and double-counting early on and a closing module on auditing SQL that an AI assistant wrote.

References

• Ganz, J., & Perigaud, B. (2026, April 7). Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update. dbt Developer Blog. Accessed 2026-09-16.
• Zayats, T. (2025, March 31). Agentic Semantic Model Improvement: Elevating Text-to-SQL Performance. Snowflake Engineering Blog. Accessed 2026-09-16.
• MotherDuck. Your Data Model Is the Semantic Layer. MotherDuck Blog (undated). Accessed 2026-09-16.
• Li, J., Hui, B., Qu, G., et al. (2023). Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs. arXiv:2305.03111 — source of the 92.96% human-performance baseline on BIRD-SQL. Accessed 2026-09-16.
• SophiArch. SQL for Data Analysis — course covering filtering, aggregation, joins, CTEs, window functions and cohort analysis, closing with a module on auditing AI-generated SQL for the silent failure modes described here: double-counting joins, NULL traps and misplaced filters.