schemagate sits between an LLM and a database. It shows the model only the tables and columns the user is allowed to read, and before any generated SQL runs, it checks that the query doesn't touch anything else.
I ran it on Spider-ACL, a dataset from an EACL 2026 paper (Klisura et al.). It takes Spider's questions, gives each one four user roles as real PostgreSQL GRANTs, and labels every question with what Postgres itself said when the reference query ran as that role: allowed or "permission denied". 19,624 question-and-role pairs. The job: answer the allowed ones, refuse the rest.
The first run was bad
A 200-question pilot scored an F1 of 0.83. It was refusing questions it should have answered.
32 of the 33 wrong refusals had the same cause, and it was mine. The check that reads table names out of a query handled FROM students as t1 fine, but not FROM students AS T1. One regex was case-sensitive. With upper-case AS, it read T1 as a second table, couldn't find a table called T1, and refused the query as "not available to this caller".
Most generated SQL writes AS in capitals. So real users would have hit this too. One flag, a regression test, and a check that the tests fail without the fix.
After the fix, all 19,624
Same pipeline, Claude Sonnet 5.5, every record:
| refusal F1 | answered a forbidden question | read data the role wasn't granted | |
|---|---|---|---|
| schemagate | 0.984 | 1.8% | 0 of 8,128 answered |
| paper's best (fine-tuned Mistral-7B) | 0.933 | not reported | not reported |
On 600 records the pilot never touched: 0.986.
The zero in the last column I didn't take on trust. I parsed every answered query with a separate SQL parser and compared each table and column against the role's GRANTs. None outside them.
The 1.8% are questions the model answered from columns it was allowed to see. Some are fine (the reference query fetched a column the question never asked for). Some are near-miss answers. That's what's left to fix.
Honest limits
- The paper's systems used 2024-25 models. This is a 2026 model, so part of the gap is the model.
- This scores the answer-or-refuse decision, not whether each answer is right.
- The paper's BIRD half isn't run yet.
What I took from it
My own tests passed. A benchmark someone else wrote, with labels from Postgres itself, found a bug in two days. If you build access control for LLMs, run it against labels you didn't make.
Fixed in schemagate 1.6.1. Every decision and query: results. Write-up: results page.
Top comments (0)