<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Ashish sinha</title>
    <description>The latest articles on DEV Community by Ashish sinha (@ashish_sinha_5241c7673d93).</description>
    <link>https://dev.to/ashish_sinha_5241c7673d93</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4114397%2F3524a3c8-07e7-4ac0-a016-fd931fa4e63d.jpg</url>
      <title>DEV Community: Ashish sinha</title>
      <link>https://dev.to/ashish_sinha_5241c7673d93</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/ashish_sinha_5241c7673d93"/>
    <language>en</language>
    <item>
      <title>One constant was costing my schema retriever 17 points</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Fri, 13 Nov 2026 13:50:44 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/one-constant-was-costing-my-schema-retriever-17-points-l19</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/one-constant-was-costing-my-schema-retriever-17-points-l19</guid>
      <description>&lt;p&gt;If your warehouse columns are called &lt;code&gt;questregistry&lt;/code&gt; and &lt;code&gt;climtune&lt;/code&gt;, keyword search picks the wrong tables, and the model writes confident SQL against them.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;schemagate&lt;/strong&gt; sends the model only the tables a question needs and the caller may read: &lt;strong&gt;91.1% fewer schema tokens&lt;/strong&gt; (median, databases with 50+ tables). &lt;strong&gt;#1 on BIRD's Mini-Dev leaderboard&lt;/strong&gt; in SQLite / PostgreSQL / MySQL (81.0 / 72.6 / 72.4 EX, official board); &lt;strong&gt;74.34 EX on BIRD's hidden test set&lt;/strong&gt;, scored by the BIRD team. Listed in LangChain's retriever docs, usable from LangGraph.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I found
&lt;/h2&gt;

&lt;p&gt;Snowflake's SQL-Schema-Retrieval benchmark (arXiv 2607.13311) asks one thing: given the question, rank this database's tables. On its hardest set, LiveSQLBench-Large (971 tables, column names that are codes), my fused ranking scored &lt;strong&gt;48.2 recall@10 while its own vector channel alone scored 60.8&lt;/strong&gt;. The fusion was losing to itself.&lt;/p&gt;

&lt;p&gt;The cause was the RRF constant everyone copies, k=60. At 60, being &lt;em&gt;anywhere&lt;/em&gt; in three keyword channels outweighs being &lt;em&gt;first&lt;/em&gt; in the vector one, so readable distractor tables that shared one word with the question took the slots.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix (1.18.0), each piece measured on all five sets before it was kept
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;RRF constant 10, so rank matters again&lt;/li&gt;
&lt;li&gt;a column channel: each table scored by its single best column (name + comment), max not sum, so a wide table can't win by breadth&lt;/li&gt;
&lt;li&gt;keyword votes scaled by how much of the question the match explains&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Recall@10, no LLM calls
&lt;/h2&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Spider&lt;/th&gt;
&lt;th&gt;BIRD&lt;/th&gt;
&lt;th&gt;BEAVER&lt;/th&gt;
&lt;th&gt;LiveSQLBench&lt;/th&gt;
&lt;th&gt;LSB-Large&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;hashed embedder (no extras)&lt;/td&gt;
&lt;td&gt;100.0&lt;/td&gt;
&lt;td&gt;99.2&lt;/td&gt;
&lt;td&gt;65.9&lt;/td&gt;
&lt;td&gt;95.3&lt;/td&gt;
&lt;td&gt;64.9 (was 48.2)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;+ all-MiniLM-L6-v2&lt;/td&gt;
&lt;td&gt;100.0&lt;/td&gt;
&lt;td&gt;99.8&lt;/td&gt;
&lt;td&gt;66.2&lt;/td&gt;
&lt;td&gt;97.6&lt;/td&gt;
&lt;td&gt;73.9 (was 51.9)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;paper's best model&lt;/td&gt;
&lt;td&gt;98.5&lt;/td&gt;
&lt;td&gt;99.1&lt;/td&gt;
&lt;td&gt;65.8&lt;/td&gt;
&lt;td&gt;91.7&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;75.8&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;What it doesn't do:&lt;/strong&gt; still under Qwen3-Embedding-4B on LSB-Large. The gap is the 22M-parameter sentence model, not the fusion: bge-small (33M) alone reaches 74.2. BIRD and BEAVER are level with the paper, not ahead. These are retrieval numbers I measured, not answer accuracy; the script is in the repo.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Also new:&lt;/strong&gt; &lt;code&gt;schemagate certify &amp;lt;url&amp;gt; --golden&lt;/code&gt; writes a question per table from &lt;em&gt;your&lt;/em&gt; schema and names the tables no question will reach. And from 1.17: the schema as a property graph (Oracle &lt;code&gt;CREATE PROPERTY GRAPH&lt;/code&gt; generated for you), concepts pulled from comments and view SQL, joins mined from the query log with every literal stripped first.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;pip install schemagate&lt;/code&gt; · &lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt; · &lt;a href="https://github.com/ashishsinha1602/schemagate/blob/main/BENCHMARKS.md#schema-retrieval-scored-by-snowflakes-benchmark" rel="noopener noreferrer"&gt;the benchmark section&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>database</category>
      <category>llm</category>
      <category>sql</category>
    </item>
    <item>
      <title>A LangGraph SQL agent that only sees the tables it needs</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Fri, 06 Nov 2026 04:15:07 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/a-langgraph-sql-agent-that-only-sees-the-tables-it-needs-23no</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/a-langgraph-sql-agent-that-only-sees-the-tables-it-needs-23no</guid>
      <description>&lt;p&gt;Most text-to-SQL agents paste the whole schema into every prompt. On a real database, most of it is tables the question never touches, and you pay for them on every call.&lt;/p&gt;

&lt;p&gt;I added a small LangGraph example to &lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;schemagate&lt;/a&gt; that does it differently. It has four nodes:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;select_schema&lt;/strong&gt;: &lt;code&gt;SchemagateRetriever&lt;/code&gt; picks the few tables the question needs. This step runs offline, with no model call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;write_sql&lt;/strong&gt;: the model writes one SELECT using only those tables.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;run_sql&lt;/strong&gt;: the agent runs the query. If it fails, the error goes back to the model once.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;answer&lt;/strong&gt;: the model answers in one sentence from the returned rows.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;On the bundled 42-table demo database:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Question&lt;/th&gt;
&lt;th&gt;Schema tokens sent&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Which customers owe us money?&lt;/td&gt;
&lt;td&gt;489 vs 2,199 (78% fewer)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;How many orders did each carrier ship?&lt;/td&gt;
&lt;td&gt;706 vs 2,199 (68% fewer)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Which products are below their reorder level, and in which warehouse?&lt;/td&gt;
&lt;td&gt;563 vs 2,199 (74% fewer)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;All three answers were correct with Claude Haiku. On databases with 50+ tables the saving reaches about 91%. The selection also respects roles: a table the caller can't read never appears in the prompt.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;langchain-schemagate langgraph langchain-anthropic tiktoken
python examples/langgraph_sql_agent.py
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;To use your own database, swap &lt;code&gt;demo_catalog()&lt;/code&gt; for &lt;code&gt;Catalog().bootstrap("postgresql://...")&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;schemagate is #1 on the BIRD Mini-Dev leaderboard (SQLite, PostgreSQL, MySQL), is listed in LangChain's official retriever integrations, and is Apache-2.0.&lt;/p&gt;

&lt;p&gt;Example: &lt;a href="https://github.com/ashishsinha1602/schemagate/blob/main/examples/langgraph_sql_agent.py" rel="noopener noreferrer"&gt;examples/langgraph_sql_agent.py&lt;/a&gt;&lt;/p&gt;

</description>
      <category>agents</category>
      <category>database</category>
      <category>llm</category>
      <category>sql</category>
    </item>
    <item>
      <title>My open-source text-to-SQL layer is #1 on BIRD Mini-Dev in all three dialects</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Sat, 31 Oct 2026 03:33:22 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/my-open-source-text-to-sql-layer-is-1-on-bird-mini-dev-in-all-three-dialects-17a</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/my-open-source-text-to-sql-layer-is-1-on-bird-mini-dev-in-all-three-dialects-17a</guid>
      <description>&lt;p&gt;BIRD Mini-Dev is 500 text-to-SQL questions over 11 databases, in SQLite, PostgreSQL and MySQL versions, scored by whether the query returns the right rows. BIRD keeps the leaderboard and checks the submissions.&lt;/p&gt;

&lt;p&gt;schemagate, the Apache-2.0 library I maintain, is now first in all three columns:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;schemagate&lt;/th&gt;
&lt;th&gt;Next entry&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;SQLite&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;81.0&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;75.6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PostgreSQL&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;72.6&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;69.0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MySQL&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;72.4&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;49.2&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The setup is plain: schemagate picks the tables a question needs, the prompt carries only those plus BIRD's column notes, and Claude Opus 5.5 writes one query. No candidate voting and no agent loop. If the query errors, the model sees the error and tries again.&lt;/p&gt;

&lt;p&gt;schemagate's main job is access control. It removes the tables the caller may not read before the model sees the schema. A benchmark doesn't test that part, but it shows the filtering doesn't cost accuracy.&lt;/p&gt;

&lt;p&gt;What it isn't: the main BIRD board, which uses a hidden test set. Our first test result there was 72.11, and a rerun is pending.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Leaderboard: &lt;a href="https://bird-bench.github.io/" rel="noopener noreferrer"&gt;https://bird-bench.github.io/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Run files and commands: &lt;a href="https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/bird_minidev" rel="noopener noreferrer"&gt;https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/bird_minidev&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;&lt;code&gt;pip install schemagate&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>sql</category>
      <category>database</category>
      <category>ai</category>
      <category>opensource</category>
    </item>
    <item>
      <title>I compared four ways to pick tables for a text-to-SQL prompt</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Wed, 28 Oct 2026 14:25:31 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/i-compared-four-ways-to-pick-tables-for-a-text-to-sql-prompt-aog</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/i-compared-four-ways-to-pick-tables-for-a-text-to-sql-prompt-aog</guid>
      <description>&lt;p&gt;Every text-to-SQL prompt carries a schema. Send all of it and you pay for every table on every call. Send a few and you risk leaving out the one the question needs. So how you pick the tables matters.&lt;/p&gt;

&lt;p&gt;I tested four ways on the same questions, with no model calls at all:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Full schema&lt;/strong&gt;: no picking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BM25&lt;/strong&gt;: keyword search over each table's definition.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Embeddings&lt;/strong&gt;: &lt;code&gt;all-MiniLM-L6-v2&lt;/code&gt; similarity, the usual way LangChain, LlamaIndex or Vanna pick tables from a vector store.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;schemagate&lt;/strong&gt;, the open-source library I maintain, using the same MiniLM model.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Each method got the same number of tables: 12 on Spider 2.0's big BigQuery and Snowflake warehouses, 6 on BIRD.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Warehouses: all needed tables picked&lt;/th&gt;
&lt;th&gt;Warehouses: tokens&lt;/th&gt;
&lt;th&gt;BIRD: all needed tables picked&lt;/th&gt;
&lt;th&gt;BIRD: tokens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Full schema&lt;/td&gt;
&lt;td&gt;93.8%&lt;/td&gt;
&lt;td&gt;56,237&lt;/td&gt;
&lt;td&gt;100%&lt;/td&gt;
&lt;td&gt;579&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BM25&lt;/td&gt;
&lt;td&gt;78.7%&lt;/td&gt;
&lt;td&gt;1,513&lt;/td&gt;
&lt;td&gt;81.0%&lt;/td&gt;
&lt;td&gt;407&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Embeddings&lt;/td&gt;
&lt;td&gt;78.2%&lt;/td&gt;
&lt;td&gt;1,555&lt;/td&gt;
&lt;td&gt;89.4%&lt;/td&gt;
&lt;td&gt;438&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;schemagate&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;85.3%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1,677&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;98.7%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;468&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On the same question, schemagate had every needed table where embeddings missed 17 times on the warehouses and was beaten once. On BIRD, 151 to 9.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What it costs:&lt;/strong&gt; 8–11% more tokens than plain retrieval, mostly the tables needed to join the ones it picked. That join step is a big part of why it finds more.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What I didn't test:&lt;/strong&gt; selectors that ask an LLM to pick tables. They cost a model call per question; this comparison is the free tier.&lt;/p&gt;

&lt;p&gt;The script and the full results are in this pull request, if you want to run it on your own questions: &lt;a href="https://github.com/ashishsinha1602/schemagate/pull/189" rel="noopener noreferrer"&gt;https://github.com/ashishsinha1602/schemagate/pull/189&lt;/a&gt;&lt;/p&gt;

</description>
      <category>database</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>sql</category>
    </item>
    <item>
      <title>schemagate is now #1 on the official BIRD Mini-Dev leaderboard</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Mon, 26 Oct 2026 02:10:49 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/schemagate-is-now-1-on-the-official-bird-mini-dev-leaderboard-4cec</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/schemagate-is-now-1-on-the-official-bird-mini-dev-leaderboard-4cec</guid>
      <description>&lt;p&gt;The BIRD team merged our entry, so it is official: &lt;strong&gt;schemagate + Claude Opus 5.5 is #1 on the BIRD Mini-Dev leaderboard&lt;/strong&gt; in all three dialects, and schemagate + Gemini 2.5 Flash is #2.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;SQLite&lt;/th&gt;
&lt;th&gt;MySQL&lt;/th&gt;
&lt;th&gt;PostgreSQL&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;schemagate + Claude Opus 5.5&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;81.00&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;72.40&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;72.60&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;schemagate + Gemini 2.5 Flash&lt;/td&gt;
&lt;td&gt;63.60&lt;/td&gt;
&lt;td&gt;54.80&lt;/td&gt;
&lt;td&gt;55.60&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Execution accuracy on BIRD's 500 Mini-Dev questions, scored by BIRD's own evaluator.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What schemagate does:&lt;/strong&gt; before the model writes SQL, it picks only the tables and columns the question needs (and only ones that caller is allowed to read), and refuses generated SQL that reaches past them. One model call per question.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Leaderboard: &lt;a href="https://bird-bench.github.io/" rel="noopener noreferrer"&gt;https://bird-bench.github.io/&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Merged entry: &lt;a href="https://github.com/bird-bench/bird-bench.github.io/pull/251" rel="noopener noreferrer"&gt;https://github.com/bird-bench/bird-bench.github.io/pull/251&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Code, every prediction and the scorer logs: &lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;https://github.com/ashishsinha1602/schemagate&lt;/a&gt; (&lt;code&gt;pip install schemagate&lt;/code&gt;)&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Thanks to the BIRD team for reviewing and merging it.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>database</category>
      <category>llm</category>
      <category>sql</category>
    </item>
    <item>
      <title>What one month of public benchmarks says about my text-to-SQL access layer</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Sat, 24 Oct 2026 03:20:24 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/what-one-month-of-public-benchmarks-says-about-my-text-to-sql-access-layer-4n91</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/what-one-month-of-public-benchmarks-says-about-my-text-to-sql-access-layer-4n91</guid>
      <description>&lt;p&gt;&lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;schemagate&lt;/a&gt; is an open-source layer for text-to-SQL that starts from one question: &lt;strong&gt;what is this caller allowed to see?&lt;/strong&gt; It shows the model only the tables the current user may read, before any SQL is written, reads the permissions from your database's own grants, and refuses a query that touches anything else before it reaches the database.&lt;/p&gt;

&lt;p&gt;This month I ran it on public benchmarks. Every number below comes from that benchmark's own evaluator, and the predictions are in the repo.&lt;/p&gt;

&lt;h2&gt;
  
  
  Access control
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;RBAC Text-to-SQL&lt;/strong&gt; (SIGMOD '27): &lt;strong&gt;90.1 AC-F1&lt;/strong&gt;, against 88.1 for the paper's best setup with the same model.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Spider-ACL&lt;/strong&gt;: &lt;strong&gt;0.984 refusal F1&lt;/strong&gt; over all 19,624 question-and-role pairs; best published 0.933.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;0 queries read data the role was not granted&lt;/strong&gt;, out of 8,128 answered.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  BIRD
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dev set, all 1,534 questions: 77.1% execution accuracy with one model call per question.&lt;/strong&gt; No candidate voting, no agent loop.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mini-Dev: 81.0 / 72.6 / 72.4&lt;/strong&gt; on SQLite / PostgreSQL / MySQL, above the current listed leaders. The leaderboard pull request is waiting on the maintainers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hidden test set:&lt;/strong&gt; submitted; the BIRD team runs it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Spider 2.0
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;SQLite questions (135): 71.9%&lt;/strong&gt; with one call per question. The best published agent on the same subset reports 57.8% &lt;a href="mailto:Pass@1"&gt;Pass@1&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;The BigQuery and Snowflake runs for a full leaderboard entry are under way.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Oracle
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;UniQL, Oracle dialect, 1,534 questions: 64.5% on the first try&lt;/strong&gt;, against a best published baseline of 49.6. Two prompt rules about Oracle syntax did most of it.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Adoption
&lt;/h2&gt;

&lt;p&gt;About &lt;strong&gt;10,000 downloads&lt;/strong&gt; on PyPI so far (pypistats, mirrors excluded). Apache-2.0. Works with Oracle, PostgreSQL, SQL Server, MySQL and SQLite, as an MCP server, a LangChain retriever or a CLI.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;schemagate
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;What these numbers are not: the Mini-Dev rows are self-reported until BIRD merges them, single runs vary by about a point, and the Spider 2.0 figure is one third of that benchmark. The details and every caveat are on the &lt;a href="https://ashishsinha1602.github.io/schemagate/benchmarks/" rel="noopener noreferrer"&gt;benchmarks page&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;GitHub:&lt;/strong&gt; &lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;https://github.com/ashishsinha1602/schemagate&lt;/a&gt; · a star helps other people find it.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>opensource</category>
      <category>security</category>
      <category>sql</category>
    </item>
    <item>
      <title>77.1% on BIRD dev with one model call per question, and a test-set submission in</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Thu, 22 Oct 2026 14:55:10 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/771-on-bird-dev-with-one-model-call-per-question-and-a-test-set-submission-in-5g6p</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/771-on-bird-dev-with-one-model-call-per-question-and-a-test-set-submission-in-5g6p</guid>
      <description>&lt;p&gt;&lt;a href="https://bird-bench.github.io/" rel="noopener noreferrer"&gt;BIRD&lt;/a&gt; is the text-to-SQL benchmark the serious systems compete on: 1,534 dev questions over 11 real databases, scored by execution accuracy, with a hidden test set that only the BIRD team can run. I ran &lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;schemagate&lt;/a&gt; on the full dev set this week and sent the test-set submission.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Dev set, all 1,534 questions, BIRD's own evaluator: 77.12% execution accuracy&lt;/strong&gt; (simple 80.97, moderate 71.34, challenging 71.03), with Claude Opus 5.5.&lt;/p&gt;

&lt;p&gt;What makes that number interesting is the cost. Per question:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;schemagate reflects the database and picks the tables the question needs;&lt;/li&gt;
&lt;li&gt;the model sees only those tables' DDL and BIRD's column notes;&lt;/li&gt;
&lt;li&gt;the model writes &lt;strong&gt;one&lt;/strong&gt; query; if SQLite errors, it sees the error and fixes it (that happened once in 1,534).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That's one model call per question, about 2,100 prompt tokens, no candidate generation, no voting, no agent loop. The systems at the top of the main leaderboard (82.95 on the test set) typically spend 5–20 calls per question. We're submitting to find out where one call lands on the hidden test set; BIRD's turnaround for API-based methods is 1–5 days, and I'll post the verified number either way.&lt;/p&gt;

&lt;p&gt;Three things worth knowing before you compare:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;A re-run of the same code gave 80.20 on Mini-Dev where the first run gave 81.00.&lt;/strong&gt; Opus is not deterministic run to run; treat single-run numbers as ±1.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BIRD's dev set has known annotation errors&lt;/strong&gt; (Jin et al., VLDB 2026 found problems in about half of Mini-Dev's reference answers). Every entry is scored against the same answers, so rankings hold, but absolute numbers are softer than they look.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The leaderboard is self-reported for dev and Mini-Dev.&lt;/strong&gt; Our Mini-Dev rows (81.0 / 72.6 / 72.4 on SQLite / PostgreSQL / MySQL, above the current #1 in each) are in &lt;a href="https://github.com/bird-bench/bird-bench.github.io/pull/251" rel="noopener noreferrer"&gt;PR #251&lt;/a&gt; awaiting the maintainers. The test set is the only part BIRD runs itself.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Why bother with a one-call system at all? Because schemagate's job isn't the leaderboard; it's making sure a caller never sees a table they can't read, and never gets a query run that touches one. On the access-control benchmarks (RBAC Text-to-SQL, Spider-ACL) it has the best published results, with zero ungranted reads in 8,128 queries. The BIRD number shows that layer costs nothing in accuracy.&lt;/p&gt;

&lt;p&gt;Predictions, evaluator output and the exact submission package: &lt;a href="https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/bird_dev" rel="noopener noreferrer"&gt;https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/bird_dev&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>database</category>
      <category>llm</category>
      <category>sql</category>
    </item>
    <item>
      <title>Text-to-SQL on Oracle: two prompt rules took a light model from 35% to 64%</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Tue, 20 Oct 2026 02:46:49 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/text-to-sql-on-oracle-two-prompt-rules-took-a-light-model-from-35-to-64-3mpc</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/text-to-sql-on-oracle-two-prompt-rules-took-a-light-model-from-35-to-64-3mpc</guid>
      <description>&lt;p&gt;&lt;a href="https://github.com/JerryGao818/UniQL" rel="noopener noreferrer"&gt;UniQL&lt;/a&gt; is a new benchmark that translates BIRD's 1,534 text-to-SQL questions into 16 SQL dialects. Its Oracle column is hard. The best published result is &lt;strong&gt;49.61%&lt;/strong&gt; (Claude Sonnet 4.5), and Gemini 2.5 Pro scores 26.14%.&lt;/p&gt;

&lt;p&gt;I ran &lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;schemagate&lt;/a&gt; on it with a light model, &lt;strong&gt;Gemini 2.5 Flash&lt;/strong&gt;. The scores below come from UniQL's own evaluator, on Oracle Free 23ai.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;UniQL Oracle, 1,534 questions&lt;/th&gt;
&lt;th&gt;First try&lt;/th&gt;
&lt;th&gt;With error repair&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;schemagate + Gemini 2.5 Flash&lt;/td&gt;
&lt;td&gt;35.0&lt;/td&gt;
&lt;td&gt;69.9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;+ two Oracle rules in the prompt&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;64.5&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;67.6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Best published (Claude Sonnet 4.5, single try)&lt;/td&gt;
&lt;td&gt;49.6&lt;/td&gt;
&lt;td&gt;—&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;The first run looked good for the wrong reason.&lt;/strong&gt; 69.9% was mostly schemagate's repair loop: run the query, show Oracle's error to the model, let it fix it. On a first try alone, Flash got 35%. That's below the bare models, and not a fair comparison with their single-try numbers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Two mistakes caused almost all of it.&lt;/strong&gt; Of the 694 failed first queries:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;551 wrote &lt;code&gt;FROM account AS a&lt;/code&gt;. Oracle doesn't allow &lt;code&gt;AS&lt;/code&gt; before a table alias.&lt;/li&gt;
&lt;li&gt;100 used a column named &lt;code&gt;DATE&lt;/code&gt; or &lt;code&gt;NUMBER&lt;/code&gt; without quotes. Both are reserved words.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;So schemagate now tells the model Oracle's rules whenever the target is Oracle. First-try accuracy went from &lt;strong&gt;35.0% to 64.5%&lt;/strong&gt;, about 15 points above the best published model, with a lighter one. Failed first queries dropped from 694 to 85. Other databases' prompts are unchanged.&lt;/p&gt;

&lt;p&gt;The lesson I'd take: measure before you tune, and report the number that matches what others report. A repair loop is a fine design, but it doesn't belong in a column of single-try results.&lt;/p&gt;

&lt;p&gt;Submitted to the UniQL leaderboard: &lt;a href="https://github.com/JerryGao818/UniQL/issues/2" rel="noopener noreferrer"&gt;JerryGao818/UniQL#2&lt;/a&gt;. Predictions, evaluator output and the workflow that reproduces it: &lt;a href="https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/uniql_oracle" rel="noopener noreferrer"&gt;https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/uniql_oracle&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>database</category>
      <category>llm</category>
      <category>sql</category>
    </item>
    <item>
      <title>schemagate tops BIRD Mini-Dev in SQLite, PostgreSQL and MySQL</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Mon, 19 Oct 2026 13:28:50 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/schemagate-tops-bird-mini-dev-in-sqlite-postgresql-and-mysql-372d</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/schemagate-tops-bird-mini-dev-in-sqlite-postgresql-and-mysql-372d</guid>
      <description>&lt;p&gt;&lt;a href="https://github.com/bird-bench/mini_dev" rel="noopener noreferrer"&gt;BIRD Mini-Dev&lt;/a&gt; is 500 text-to-SQL questions over 11 real databases, in three dialects. It is scored by execution accuracy: run the query, and compare its rows with the gold query's rows. I ran &lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;schemagate&lt;/a&gt; on all of it. Every number below comes from BIRD's own evaluator.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;schemagate + Claude Opus 5.5&lt;/th&gt;
&lt;th&gt;#1 until now&lt;/th&gt;
&lt;th&gt;that entry's model&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;SQLite&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;81.0&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;75.6 (Jitto Build)&lt;/td&gt;
&lt;td&gt;Claude Opus 5.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;PostgreSQL&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;72.6&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;65.8 (Ontology2SQL)&lt;/td&gt;
&lt;td&gt;deepseek-v4-flash, as a 12-step agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;MySQL&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;72.4&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;49.2 (TA + GPT-4)&lt;/td&gt;
&lt;td&gt;GPT-4&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;SQLite is the like-for-like result.&lt;/strong&gt; The previous #1 uses the same model (per JamLabs' own LiveSQLBench listing), and schemagate is 5.4 points ahead.&lt;/p&gt;

&lt;p&gt;I also ran it with light models, because a benchmark that only a frontier model wins says little about the tool:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;SQLite&lt;/th&gt;
&lt;th&gt;PostgreSQL&lt;/th&gt;
&lt;th&gt;MySQL&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;schemagate + Gemini 2.5 Flash&lt;/td&gt;
&lt;td&gt;63.6&lt;/td&gt;
&lt;td&gt;55.6&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;54.8&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;schemagate + Claude Haiku 5.5&lt;/td&gt;
&lt;td&gt;61.8&lt;/td&gt;
&lt;td&gt;55.0&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;54.4&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On MySQL, schemagate with a light model still beats the previous best (GPT-4, 49.2). On PostgreSQL it doesn't. A light model running as an agent (65.8) beats a light model writing one query.&lt;/p&gt;

&lt;p&gt;What schemagate does per question:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;picks the tables the question needs;&lt;/li&gt;
&lt;li&gt;gives the model only their DDL and BIRD's column notes;&lt;/li&gt;
&lt;li&gt;runs the query, and shows the model any error to fix.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;There is no voting across runs, and nothing reads the gold SQL. The leaderboard PR is &lt;a href="https://github.com/bird-bench/bird-bench.github.io/pull/251" rel="noopener noreferrer"&gt;bird-bench.github.io#251&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Caveats:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;These are self-reported, as all Mini-Dev entries are.&lt;/li&gt;
&lt;li&gt;BIRD has been public since 2023, so any LLM may have seen it.&lt;/li&gt;
&lt;li&gt;The MySQL evaluator was changed only to connect over TCP on Windows.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Predictions, evaluator output and code: &lt;a href="https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/bird_minidev" rel="noopener noreferrer"&gt;https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/bird_minidev&lt;/a&gt;&lt;/p&gt;

</description>
      <category>claude</category>
      <category>database</category>
      <category>mysql</category>
      <category>sql</category>
    </item>
    <item>
      <title>A benchmark found a bug in my access-control check. After the fix it scored 0.984.</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Sat, 17 Oct 2026 14:29:53 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/a-benchmark-found-a-bug-in-my-access-control-check-after-the-fix-it-scored-0984-1fnj</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/a-benchmark-found-a-bug-in-my-access-control-check-after-the-fix-it-scored-0984-1fnj</guid>
      <description>&lt;p&gt;schemagate sits between an LLM and a database. It shows the model only the tables and columns the user is allowed to read, and before any generated SQL runs, it checks that the query doesn't touch anything else.&lt;/p&gt;

&lt;p&gt;I ran it on Spider-ACL, a dataset from an EACL 2026 paper (&lt;a href="https://aclanthology.org/2026.findings-eacl.316/" rel="noopener noreferrer"&gt;Klisura et al.&lt;/a&gt;). It takes Spider's questions, gives each one four user roles as real PostgreSQL GRANTs, and labels every question with what Postgres itself said when the reference query ran as that role: allowed or "permission denied". 19,624 question-and-role pairs. The job: answer the allowed ones, refuse the rest.&lt;/p&gt;

&lt;h2&gt;
  
  
  The first run was bad
&lt;/h2&gt;

&lt;p&gt;A 200-question pilot scored an F1 of 0.83. It was refusing questions it should have answered.&lt;/p&gt;

&lt;p&gt;32 of the 33 wrong refusals had the same cause, and it was mine. The check that reads table names out of a query handled &lt;code&gt;FROM students as t1&lt;/code&gt; fine, but not &lt;code&gt;FROM students AS T1&lt;/code&gt;. One regex was case-sensitive. With upper-case &lt;code&gt;AS&lt;/code&gt;, it read &lt;code&gt;T1&lt;/code&gt; as a second table, couldn't find a table called T1, and refused the query as "not available to this caller".&lt;/p&gt;

&lt;p&gt;Most generated SQL writes &lt;code&gt;AS&lt;/code&gt; in capitals. So real users would have hit this too. One flag, a regression test, and a check that the tests fail without the fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  After the fix, all 19,624
&lt;/h2&gt;

&lt;p&gt;Same pipeline, Claude Sonnet 5.5, every record:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;refusal F1&lt;/th&gt;
&lt;th&gt;answered a forbidden question&lt;/th&gt;
&lt;th&gt;read data the role wasn't granted&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;schemagate&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.984&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;1.8%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0 of 8,128 answered&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;paper's best (fine-tuned Mistral-7B)&lt;/td&gt;
&lt;td&gt;0.933&lt;/td&gt;
&lt;td&gt;not reported&lt;/td&gt;
&lt;td&gt;not reported&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On 600 records the pilot never touched: 0.986.&lt;/p&gt;

&lt;p&gt;The zero in the last column I didn't take on trust. I parsed every answered query with a separate SQL parser and compared each table and column against the role's GRANTs. None outside them.&lt;/p&gt;

&lt;p&gt;The 1.8% are questions the model answered from columns it was allowed to see. Some are fine (the reference query fetched a column the question never asked for). Some are near-miss answers. That's what's left to fix.&lt;/p&gt;

&lt;h2&gt;
  
  
  Honest limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;The paper's systems used 2024-25 models. This is a 2026 model, so part of the gap is the model.&lt;/li&gt;
&lt;li&gt;This scores the answer-or-refuse decision, not whether each answer is right.&lt;/li&gt;
&lt;li&gt;The paper's BIRD half isn't run yet.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What I took from it
&lt;/h2&gt;

&lt;p&gt;My own tests passed. A benchmark someone else wrote, with labels from Postgres itself, found a bug in two days. If you build access control for LLMs, run it against labels you didn't make.&lt;/p&gt;

&lt;p&gt;Fixed in schemagate 1.6.1. Every decision and query: &lt;a href="https://github.com/ashishsinha1602/schemagate/tree/main/benchmarks/results/acl" rel="noopener noreferrer"&gt;results&lt;/a&gt;. Write-up: &lt;a href="https://ashishsinha1602.github.io/schemagate/rbac-benchmark/" rel="noopener noreferrer"&gt;results page&lt;/a&gt;.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>postgres</category>
      <category>security</category>
    </item>
    <item>
      <title>I ran my text-to-SQL access control on a benchmark I didn't write</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Sat, 17 Oct 2026 02:46:35 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/i-ran-my-text-to-sql-access-control-on-a-benchmark-i-didnt-write-23n4</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/i-ran-my-text-to-sql-access-control-on-a-benchmark-i-didnt-write-23n4</guid>
      <description>&lt;p&gt;Every number I had published for schemagate came from my own tests. This month a SIGMOD'27 paper released a public benchmark for exactly the problem schemagate exists for: &lt;a href="https://arxiv.org/abs/2607.22115" rel="noopener noreferrer"&gt;Benchmarking Text-to-SQL under Role-Based Access Control&lt;/a&gt;. So I ran it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The test
&lt;/h2&gt;

&lt;p&gt;Each question comes with a role and a list of the columns that role may read. If the answer needs a column the role can't see, the right reply is "Sorry, I cannot answer." The authors score it with their own evaluator. I didn't change a line of it.&lt;/p&gt;

&lt;p&gt;Their main finding is uncomfortable for tools like mine. If you just hide the restricted columns, models still answer. They patch something together from the columns that are left. On BIRD, their best "hide the schema" setup still answered 27% of the questions it should have refused.&lt;/p&gt;

&lt;h2&gt;
  
  
  What schemagate does differently
&lt;/h2&gt;

&lt;p&gt;Three things happen for every question:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The role's policy becomes visibility. The model never sees a hidden table or column, not even its name.&lt;/li&gt;
&lt;li&gt;One line tells the model that its view is partial, without saying what is missing. If the question needs data it can't see, it should refuse instead of improvising.&lt;/li&gt;
&lt;li&gt;Every query passes a scope check before it runs. SQL that touches anything outside the policy is refused.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Results
&lt;/h2&gt;

&lt;p&gt;Same model as the paper's best BIRD run (Claude Sonnet 4.5), five trials, the authors' scoring:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;AC-F1&lt;/th&gt;
&lt;th&gt;answered when it should refuse&lt;/th&gt;
&lt;th&gt;SQL that read forbidden data&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;BIRD, schemagate&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;90.1&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;5.4%&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;BIRD, paper's best&lt;/td&gt;
&lt;td&gt;88.1&lt;/td&gt;
&lt;td&gt;7.4%&lt;/td&gt;
&lt;td&gt;not reported&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spider, schemagate&lt;/td&gt;
&lt;td&gt;93.0&lt;/td&gt;
&lt;td&gt;5.1%&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spider, paper with the same model&lt;/td&gt;
&lt;td&gt;93.0&lt;/td&gt;
&lt;td&gt;4.6%&lt;/td&gt;
&lt;td&gt;not reported&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Spider, paper's best (DeepSeek V3.2-Reasoning)&lt;/td&gt;
&lt;td&gt;95.9&lt;/td&gt;
&lt;td&gt;2.0%&lt;/td&gt;
&lt;td&gt;not reported&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;On BIRD it beats the best of the paper's thirteen models. On Spider it ties the same model and is behind the paper's best Spider result (95.9, DeepSeek V3.2-Reasoning). In neither did any query read data the role wasn't allowed to see. The scope check makes that a guarantee rather than a measured rate.&lt;/p&gt;

&lt;p&gt;And the paper's best setup shows the model the &lt;em&gt;whole&lt;/em&gt; schema plus the policy. schemagate gets these numbers without the model ever seeing what is restricted.&lt;/p&gt;

&lt;h2&gt;
  
  
  What each piece is worth
&lt;/h2&gt;

&lt;p&gt;Taking the scope check out doubled the bad answers on BIRD (5.4% to 11.3%). The one-line notice was tested on 100 questions: 16% bad answers without it, 5% with it.&lt;/p&gt;

&lt;p&gt;One surprise: giving the model BIRD's column descriptions made things &lt;strong&gt;worse&lt;/strong&gt;. Better notes on the visible columns helped it find workarounds for questions it should have refused.&lt;/p&gt;

&lt;h2&gt;
  
  
  The honest caveats
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;I chose the notice's wording on 100 BIRD questions that are part of the test set. Spider played no part in any choice, so it's the cleaner of the two results.&lt;/li&gt;
&lt;li&gt;One model, where the paper tested thirteen.&lt;/li&gt;
&lt;li&gt;My BIRD databases are the 2024 release; the paper used the 2025 one.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The notice ships on by default in schemagate 1.6.0. To reproduce the runs: &lt;code&gt;benchmarks/rbac_text2sql.py&lt;/code&gt;, plus the evaluator's reports in &lt;code&gt;benchmarks/results/rbac/&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;pip install schemagate&lt;/code&gt; · &lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>database</category>
      <category>llm</category>
      <category>sql</category>
    </item>
    <item>
      <title>My LLM reranker did nothing, and every test passed</title>
      <dc:creator>Ashish sinha</dc:creator>
      <pubDate>Fri, 16 Oct 2026 22:00:29 +0000</pubDate>
      <link>https://dev.to/ashish_sinha_5241c7673d93/my-llm-reranker-did-nothing-and-every-test-passed-20am</link>
      <guid>https://dev.to/ashish_sinha_5241c7673d93/my-llm-reranker-did-nothing-and-every-test-passed-20am</guid>
      <description>&lt;p&gt;schemagate picks the few tables a text-to-SQL model should see out of hundreds. It ranks them with plain maths first (BM25, embeddings, name matching, fused with RRF). Then, if you give it an API key, a model can reorder the top 20. That second step is the reranker.&lt;/p&gt;

&lt;p&gt;I was asked why it uses RRF and no LLM reranking. I said it has one, &lt;code&gt;--rerank&lt;/code&gt;, and went to show it working. It didn't work.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I saw
&lt;/h2&gt;

&lt;p&gt;I ran the same three questions with and without &lt;code&gt;--rerank&lt;/code&gt; on claude-haiku-5-5. The table order came out identical all three times.&lt;/p&gt;

&lt;p&gt;That can happen if the maths already got it right. So I wrapped the provider to log every call and what came back. The model was being called. It just never gave an answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  The bug
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;reply&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;provider&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;complete&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SYSTEM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prompt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;120&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;120 tokens is plenty for a reply like &lt;code&gt;3, 1, 7, 2, 5&lt;/code&gt;. It was plenty when I wrote it.&lt;/p&gt;

&lt;p&gt;Claude 5 models think before they answer, and that thinking counts against &lt;code&gt;max_tokens&lt;/code&gt;. They used up all 120 tokens thinking and stopped before writing a single number. The provider raised an error. The reranker caught it, as designed:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="nb"&gt;Exception&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;list&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# never worse than the maths alone
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;and handed back the maths order. No crash, no warning, just the same result you'd get without &lt;code&gt;--rerank&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the tests didn't catch it
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;tests/test_rerank.py&lt;/code&gt; had good tests. A model that errors leaves the order alone. A model that replies with garbage leaves the order alone. A model can't promote a table the caller isn't allowed to see. All true, all passing.&lt;/p&gt;

&lt;p&gt;Every one of them tested that failing is safe. None tested that the model actually got to answer. When the fallback is designed to be invisible, total failure looks exactly like success.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;max_tokens&lt;/code&gt; is now 1024. The test suite now has a fake model that refuses to answer below 512 tokens, so if anyone lowers it again, a test fails. And the benchmark now logs whether the model answered on each question, not just the final score.&lt;/p&gt;

&lt;p&gt;I checked it on the released package, not my working copy: 1.4.4 from PyPI got 1 of 3 model answers and changed 0 of 3 orders; 1.5.0 got 3 of 3 answers and changed 2 orders.&lt;/p&gt;

&lt;h2&gt;
  
  
  Was it worth fixing? The numbers
&lt;/h2&gt;

&lt;p&gt;Once it actually ran, I measured it for the first time on public benchmarks. Every database from Spider dev (876 tables) and BIRD dev in one pool, no hint which database a question belongs to. A question counts as a hit if every table the correct SQL uses is in the top 5.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;set     questions   maths order   reranked   only rerank right   only maths right
spider  1034        80.3%         96.5%      169                 1
bird    1534        88.6%         96.7%      125                 1
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Paired McNemar test, p around 1e-49 on Spider and 1e-36 on BIRD. It's the largest single gain in the benchmarks file.&lt;/p&gt;

&lt;p&gt;What this measures: the right tables reaching the prompt, not whether the final SQL was correct. One model. Spider and BIRD are public, so the model may have seen them in training. It only reorders tables the maths already shortlisted, and it sees table names and short summaries, not the answer SQL.&lt;/p&gt;

&lt;h2&gt;
  
  
  The lesson I'm keeping
&lt;/h2&gt;

&lt;p&gt;If you build a "safe fallback", also log when it fires. Otherwise the feature can be 100% broken and your dashboard still looks fine.&lt;/p&gt;

&lt;p&gt;Try it: &lt;code&gt;pip install "schemagate[ai]"&lt;/code&gt;, then &lt;code&gt;schemagate demo "which customers owe us money" --rerank --provider anthropic --model claude-haiku-5-5&lt;/code&gt;. In the MCP server, set &lt;code&gt;SCHEMAGATE_MCP_RERANK=1&lt;/code&gt;. Code and full benchmark: &lt;a href="https://github.com/ashishsinha1602/schemagate" rel="noopener noreferrer"&gt;https://github.com/ashishsinha1602/schemagate&lt;/a&gt;&lt;/p&gt;

</description>
      <category>debugging</category>
      <category>llm</category>
      <category>python</category>
    </item>
  </channel>
</rss>
