Every text-to-SQL benchmark score you've seen was measured without access control

Spider, BIRD, LiveSQLBench. If you have evaluated a text-to-SQL system in the
last five years you have quoted a number from one of them. All three ask the
same question: given a schema and an English question, does the system
produce SQL that returns the right rows?

None of them ask who is asking.

Every score you have seen was produced by a system with unrestricted read
access to the entire database. That is not how anyone runs one in production,
and a paper accepted to SIGMOD 2027 has now measured what happens when you
close the gap.

The paper

Benchmarking Text-to-SQL under Role-Based Access Control, by Yang Fei,
Yangfan Jiang, Yin Yang and Xiaokui Xiao (arXiv, July 2026). They take the
three benchmarks above and add what production has and benchmarks don't:
roles, and policies attached to them.

The scale of the augmentation:

• 53 databases, 399 tables, 3,353 columns
• 21,502 role-annotated query instances
• policies at column-operation granularity — not "can this role read this table" but "can this role SELECT this column"
• roles synthesised per database by an LLM-assisted pipeline, including scoped administrator roles

Then they score existing systems against it. From the abstract: many
high-performing systems, open-weight LLMs especially, "show sharp performance
degradation once access constraints are in place, due to frequent RBAC
violations."

I want to be careful here, because I have not run this myself and the
benchmark is not released yet — the paper says the pipeline, toolkit and
datasets are coming to a public repository. What follows is the mechanism
behind that sentence, which is the part I do have direct experience of.

The metric that matters

The paper's sharpest contribution is not the dataset. It's a category of
failure their metrics are built to expose, which they call RBAC-rejected
successes: SQL that is judged correct under normal evaluation and
violates the access policy.

Sit with that for a second, because it is the whole problem in five words.
Under Spider's or BIRD's grading, that query passes. It answers the question.
It returns the right rows. It is also a query the person who asked was never
permitted to run. Your evaluation harness scored it as a win.

This is why the degradation is sharp rather than gradual. It isn't that
access control makes the SQL-writing task harder. It's that a metric which
never looked at authorisation was quietly counting violations as successes,
and once you look, they move columns.

Why this happens, architecturally

Here is the part I have spent this year on, and the reason the paper's result
did not surprise me.

Every NL2SQL stack has a schema-selection step. It has to: a real schema is
hundreds or thousands of objects and you cannot put all of it in a prompt.
So something narrows the candidates before the model writes anything.

question ──► select tables ──► model writes SQL ──► execute
                                                       ↑
                                          RLS / VPD / grants act HERE

Your database's access control is at the far right. It acts on execution. But
selection happened at the far left, and it was blind — a ranker matching
words against a catalogue, with no idea who is asking.

So the model gets handed hr_compensation because a support agent's question
happened to score near it. The model does its job perfectly and writes correct
SQL against the table it was shown. Then row-level security does its job
perfectly and filters every row.

And the user sees:

No records found.

Which is a wrong answer wearing the costume of an empty result. Nothing
errored. Nothing was logged as a denial. The agent will say "there are no
compensation records matching that" with total confidence, and the person
reading it has no way to distinguish "the data doesn't exist" from "you
aren't allowed to see it." Those are very different sentences and your stack
just collapsed them into one.

The variant that should worry you more: the schema itself is information. A
table named hr_compensation_2027_layoffs tells the reader something real,
even when zero rows come back. Putting that name in a prompt is a disclosure,
and RLS cannot retract it, because RLS filters rows and the leak was in the
DDL.

What the paper does not do

It does not propose a system. It is a benchmark and an evaluation
methodology, deliberately. Read that as good news: a top-tier venue has
named and measured the problem, and left the engineering open.

The obvious fix — "just re-check permissions after the SQL comes back" —
doesn't hold, for two reasons. It's too late for the DDL disclosure above.
And a post-hoc check can only tell you the query was disallowed; it cannot
produce the answer the caller was entitled to, because the objects that
would have answered it were never candidates.

The fix has to be at selection. Whatever narrows hundreds of objects to six
has to know who is asking:

sel = catalog.select(
    "salary by employee",
    principal=Principal("okta:jdoe", roles={"analyst"}),
)
sel.table_names        # hr_compensation is not here
sel.prompt_fragment()  # and its name is not in the text the model sees

Not ranked last. Absent. The difference matters: a table ranked last is still
in the candidate set, still one prompt-budget change away from being included,
and still named in your logs.

Two properties worth insisting on if you build this yourself:

A restricted object and a nonexistent one must be indistinguishable.
Otherwise "access denied" versus "no such table" leaks the schema one probe at
a time.

Scoping is not authentication. If your selector takes a principal, then
whatever hands it that principal is now a trust boundary. A component that
accepts principal="okta:admin" from an untrusted caller has given away
everything. Scoping decides what an authenticated identity may see; it does
not establish the identity.

What I'd like to be able to tell you

I maintain a library that does exactly the selection step above, and the
honest position is that I have self-reported numbers and no third-party ones.
That is worth precisely as much as you'd expect.

This benchmark is the fix for that, and it's the reason I'm writing about a
paper rather than a release. When the authors publish the pipeline, I will run
it and post the results — including if they are bad, which is a promise that
costs nothing to make and something to keep, so hold me to it.

Until then, the useful takeaway isn't about any library. It's this: if you
are evaluating a text-to-SQL system, your benchmark score was measured with
God-mode access to the database, and the number you are about to put in a
slide does not describe how the system behaves for your actual users. The
gap between those two things has now been measured by people with no product
to sell, and it is large.

---

The paper: Benchmarking Text-to-SQL under Role-Based Access Control,
Fei, Jiang, Yang & Xiao, SIGMOD 2027. The base benchmarks:
BIRD, LiveSQLBench, Spider.

The selection library is schemagate,
Apache-2.0, with a browser demo at
ashishsinha1602.github.io/schemagate
that runs the real selector client-side — flip the caller's roles and watch
the restricted table leave the prompt.

Comments

[kevinbai]

The "RBAC-rejected success" framing deserves to spread beyond text-to-SQL — it's the same shape as agent benchmarks that score a tool call as correct without checking whether the caller was still authorised at execution time. The indistinguishability property is the key insight: "ranked last" still leaks existence through logs and prompt-budget drift, only "absent" is safe. Schema selection taking a principal is going to be table stakes for any NL2SQL stack that sells into regulated industries.