Every text-to-SQL demo is flawless. Every production rollout is a coin flip.
The model didn't get worse between the demo and your warehouse. Your schema crossed a threshold it cannot reason past.
Accuracy doesn't decay. It collapses.
The intuition most teams carry is that accuracy degrades gradually as queries get harder. It doesn't. It holds, holds, holds — then falls off a cliff the moment a question needs a join the model has to guess at.
On our own enterprise benchmark, raw schema access scored 14.5%. The same model, given compiled and governed context, scored 98.2%.
Same model. Same questions. The only variable was the context it ran on.
The three gaps that create the cliff
| Gap | What the model sees | What it needs |
|---|---|---|
| Naming |
cust_t, cust_type, customer_seg
|
One typed entity with synonyms |
| Joins | 200 tables, 3 plausible paths | A proven path, or a hard failure |
| Rules | No sign of the fiscal calendar | Grain and filters attached to the metric |
Any one of these produces syntactically perfect SQL that returns the wrong number. That is the dangerous failure — a query that errors gets fixed, a query that returns 8% instead of 12% gets pasted into a board deck.
Why "temperature: 0" is not determinism
Setting temperature to zero makes sampling greedy, not correct. Greedy decoding over an ambiguous schema reliably picks the same wrong join every time. You have traded variance for consistent error, which is worse — now it looks trustworthy.
Determinism has to come from architecture, not decoding parameters:
- The LLM parses language into a typed intent
- The semantic graph — not the model — resolves entities, metrics and grain
- The planner proves a join path exists, or compilation fails
- Policy predicates are injected before SQL is emitted
Ambiguity produces a refusal instead of a guess. That is the whole difference.
The full breakdown — what the famous benchmark numbers actually measure, what failure looks like in real SQL, what a wrong number costs, and how every serious vendor has converged on the same answer — is here:
👉 The Text-to-SQL Accuracy Cliff: Why Deterministic Compilers Beat LLM Guessing
Originally published at colrows.com/blogs/text-to-sql-accuracy-cliff
Top comments (0)