DEV Community

Cover image for Text-to-SQL on Oracle: 63.7% first try, on a benchmark my bug reports helped fix
Ashish sinha
Ashish sinha

Posted on

Text-to-SQL on Oracle: 63.7% first try, on a benchmark my bug reports helped fix

Most text-to-SQL prompts carry the whole schema. schemagate sends only the tables the question needs: 91% fewer schema tokens on 50+ table databases. It is #1 on BIRD's Mini-Dev leaderboard in SQLite, PostgreSQL and MySQL (81.0 / 72.6 / 72.4), and works as a LangChain and LangGraph retriever.

Oracle is where text-to-SQL breaks. On UniQL, BIRD's 1,534 questions translated into 16 dialects, schemagate with Gemini 3.8 Flash scores 63.69% on the first try (64.93% with repair). The best published Oracle score is 49.61% (Claude 4.5 Sonnet).

What got it there

  • Two Oracle rules in the prompt (no AS before a table alias; quote reserved words) took the first try from 35% to 64%.
  • Names stored in lower case are shown to the model in the double quotes Oracle needs.

What running it found

  • Four gold queries that error.
  • An evaluator that upper-cased string literals.
  • A loader whose users Oracle treats as system schemas.

The UniQL team fixed all of it and re-released on Nov 12; this score is on the fixed release.

Limits: Oracle only. The published baselines predate the fix and haven't been recomputed. Our leaderboard row is pending.

Results and reproduction: https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/uniql_oracle ยท pip install schemagate

Top comments (0)