BIRD Mini-Dev is 500 text-to-SQL questions over 11 real databases, in three dialects. It is scored by execution accuracy: run the query, and compare its rows with the gold query's rows. I ran schemagate on all of it. Every number below comes from BIRD's own evaluator.
| schemagate + Claude Opus 5.5 | #1 until now | that entry's model | |
|---|---|---|---|
| SQLite | 81.0 | 75.6 (Jitto Build) | Claude Opus 5.5 |
| PostgreSQL | 72.6 | 65.8 (Ontology2SQL) | deepseek-v4-flash, as a 12-step agent |
| MySQL | 72.4 | 49.2 (TA + GPT-4) | GPT-4 |
SQLite is the like-for-like result. The previous #1 uses the same model (per JamLabs' own LiveSQLBench listing), and schemagate is 5.4 points ahead.
I also ran it with light models, because a benchmark that only a frontier model wins says little about the tool:
| SQLite | PostgreSQL | MySQL | |
|---|---|---|---|
| schemagate + Gemini 2.5 Flash | 63.6 | 55.6 | 54.8 |
| schemagate + Claude Haiku 5.5 | 61.8 | 55.0 | 54.4 |
On MySQL, schemagate with a light model still beats the previous best (GPT-4, 49.2). On PostgreSQL it doesn't. A light model running as an agent (65.8) beats a light model writing one query.
What schemagate does per question:
- picks the tables the question needs;
- gives the model only their DDL and BIRD's column notes;
- runs the query, and shows the model any error to fix.
There is no voting across runs, and nothing reads the gold SQL. The leaderboard PR is bird-bench.github.io#251.
Caveats:
- These are self-reported, as all Mini-Dev entries are.
- BIRD has been public since 2023, so any LLM may have seen it.
- The MySQL evaluator was changed only to connect over TCP on Windows.
Predictions, evaluator output and code: https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/bird_minidev
Top comments (0)