Prompt engineering improves text-to-SQL accuracy mainly by supplying schema context, vetted examples, and execution feedback—not by phrasing the question…
TL;DR: The accuracy gap is a schema problem before it’s a language problem: on the BIRD benchmark, human accuracy sits near 93%, the leading published system reaches roughly 82%, and on Spider 2.0’s enterprise schemas (800+ columns), a bare model’s success rate can fall to 10–20%. The number looks reasonable. The query ran, returned a result, and looked exactly like every correct query before it.
Key takeaways
- Schema size makes the picture worse, not better, once you leave the benchmark.
- The lesson isn’t that the model got worse.
- The number is right.
BIRD benchmark analysis—execution accuracy, hint dependency, annotation error rate: beancount.io research log 2.
Read the full article: https://www.bestprompt.art/prompt-engineering-for-sql/
Top comments (0)