Originally published on AI Tech Connect.
The benchmark lie For a few years, text-to-SQL looked like a solved problem. Leaderboards for Spider 1.0 — the academic benchmark that defined the field — crept past 85% execution accuracy, demo videos showed analysts typing questions in plain English, and every data platform grew a "chat with your data" button. Then in November 2024 the XLang Lab released Spider 2.0, a benchmark built from real enterprise workflows instead of toy schemas, and the field got a correction it is still digesting. The headline result: agent frameworks built on GPT-4o-class models that solved 86.6% of Spider 1.0 managed just 6.0% of Spider 2.0 tasks at launch. Even o1-preview, the strongest reasoning model of the day, dropped from 91.2% to 21.3%. What changed was not the models. It was the questions. Spider 2.0…
Top comments (0)