UniQL is a new benchmark that translates BIRD's 1,534 text-to-SQL questions into 16 SQL dialects. Its Oracle column is hard. The best published result is 49.61% (Claude Sonnet 4.5), and Gemini 2.5 Pro scores 26.14%.
I ran schemagate on it with a light model, Gemini 2.5 Flash. The scores below come from UniQL's own evaluator, on Oracle Free 23ai.
| UniQL Oracle, 1,534 questions | First try | With error repair |
|---|---|---|
| schemagate + Gemini 2.5 Flash | 35.0 | 69.9 |
| + two Oracle rules in the prompt | 64.5 | 67.6 |
| Best published (Claude Sonnet 4.5, single try) | 49.6 | — |
The first run looked good for the wrong reason. 69.9% was mostly schemagate's repair loop: run the query, show Oracle's error to the model, let it fix it. On a first try alone, Flash got 35%. That's below the bare models, and not a fair comparison with their single-try numbers.
Two mistakes caused almost all of it. Of the 694 failed first queries:
- 551 wrote
FROM account AS a. Oracle doesn't allowASbefore a table alias. - 100 used a column named
DATEorNUMBERwithout quotes. Both are reserved words.
So schemagate now tells the model Oracle's rules whenever the target is Oracle. First-try accuracy went from 35.0% to 64.5%, about 15 points above the best published model, with a lighter one. Failed first queries dropped from 694 to 85. Other databases' prompts are unchanged.
The lesson I'd take: measure before you tune, and report the number that matches what others report. A repair loop is a fine design, but it doesn't belong in a column of single-try results.
Submitted to the UniQL leaderboard: JerryGao818/UniQL#2. Predictions, evaluator output and the workflow that reproduces it: https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/uniql_oracle
Top comments (2)
첫 시도 점수와 오류 수정 후 점수를 분리한 표가 비교 조건을 이해하는 데 도움이 되네요. 두 규칙을 각각 적용한 결과와 함께 적용한 결과도 나눠 보면, 어떤 오류 유형이 줄었는지 더 선명해질 것 같습니다. 프롬프트를 조정할 때 쓴 문제와 별도로 남겨둔 평가 문제의 구분도 궁금합니다.
Oracle 不让在表别名前面加 AS 这个坑我也踩过，当时排查了半天，以为是列名的问题。把方言规则显式写进 prompt 确实比 repair 循环划算，一次 repair 循环至少多一轮模型调用和错误回显。好奇问下，schemagate 有没有考虑把这类规则做成方言模板，按 target dialect 自动注入，而不是在每个 prompt 里手写？