Every text-to-SQL benchmark score you've seen was measured without access control

Spider, BIRD, LiveSQLBench. If you have evaluated a text-to-SQL system in the
last five years you have quoted a number from one of them. All three ask the
same question: given a schema and an English question, does the system
produce SQL that returns the right rows?

None of them ask who is asking.

Every score you have seen was produced by a system with unrestricted read
access to the entire database. That is not how anyone runs one in production,
and a paper accepted to SIGMOD 2027 has now measured what happens when you
close the gap.

The paper

Benchmarking Text-to-SQL under Role-Based Access Control, by Yang Fei,
Yangfan Jiang, Yin Yang and Xiaokui Xiao (arXiv, July 2026). They take the
three benchmarks above and add what production has and benchmarks don't:
roles, and policies attached to them.

The scale of the augmentation:

• 53 databases, 399 tables, 3,353 columns
• 21,502 role-annotated query instances
• policies at column-operation granularity — not "can this role read this table" but "can this role SELECT this column"
• roles synthesised per database by an LLM-assisted pipeline, including scoped administrator roles

Then they score existing systems against it. From the abstract: many
high-performing systems, open-weight LLMs especially, "show sharp performance
degradation once access constraints are in place, due to frequent RBAC
violations."

I want to be careful here, because I have not run this myself and the
benchmark is not released yet — the paper says the pipeline, toolkit and
datasets are coming to a public repository. What follows is the mechanism
behind that sentence, which is the part I do have direct experience of.

The metric that matters

The paper's sharpest contribution is not the dataset. It's a category of
failure their metrics are built to expose, which they call RBAC-rejected
successes: SQL that is judged correct under normal evaluation and
violates the access policy.

Sit with that for a second, because it is the whole problem in five words.
Under Spider's or BIRD's grading, that query passes. It answers the question.
It returns the right rows. It is also a query the person who asked was never
permitted to run. Your evaluation harness scored it as a win.

This is why the degradation is sharp rather than gradual. It isn't that
access control makes the SQL-writing task harder. It's that a metric which
never looked at authorisation was quietly counting violations as successes,
and once you look, they move columns.

Why this happens, architecturally

Here is the part I have spent this year on, and the reason the paper's result
did not surprise me.

Every NL2SQL stack has a schema-selection step. It has to: a real schema is
hundreds or thousands of objects and you cannot put all of it in a prompt.
So something narrows the candidates before the model writes anything.

question ──► select tables ──► model writes SQL ──► execute
                                                       ↑
                                          RLS / VPD / grants act HERE

Your database's access control is at the far right. It acts on execution. But
selection happened at the far left, and it was blind — a ranker matching
words against a catalogue, with no idea who is asking.

So the model gets handed hr_compensation because a support agent's question
happened to score near it. The model does its job perfectly and writes correct
SQL against the table it was shown. Then row-level security does its job
perfectly and filters every row.

And the user sees:

No records found.

Which is a wrong answer wearing the costume of an empty result. Nothing
errored. Nothing was logged as a denial. The agent will say "there are no
compensation records matching that" with total confidence, and the person
reading it has no way to distinguish "the data doesn't exist" from "you
aren't allowed to see it." Those are very different sentences and your stack
just collapsed them into one.

The variant that should worry you more: the schema itself is information. A
table named hr_compensation_2027_layoffs tells the reader something real,
even when zero rows come back. Putting that name in a prompt is a disclosure,
and RLS cannot retract it, because RLS filters rows and the leak was in the
DDL.

What the paper does not do

It does not propose a system. It is a benchmark and an evaluation
methodology, deliberately. Read that as good news: a top-tier venue has
named and measured the problem, and left the engineering open.

The obvious fix — "just re-check permissions after the SQL comes back" —
doesn't hold, for two reasons. It's too late for the DDL disclosure above.
And a post-hoc check can only tell you the query was disallowed; it cannot
produce the answer the caller was entitled to, because the objects that
would have answered it were never candidates.

The fix has to be at selection. Whatever narrows hundreds of objects to six
has to know who is asking:

sel = catalog.select(
    "salary by employee",
    principal=Principal("okta:jdoe", roles={"analyst"}),
)
sel.table_names        # hr_compensation is not here
sel.prompt_fragment()  # and its name is not in the text the model sees

Not ranked last. Absent. The difference matters: a table ranked last is still
in the candidate set, still one prompt-budget change away from being included,
and still named in your logs.

Two properties worth insisting on if you build this yourself:

A restricted object and a nonexistent one must be indistinguishable.
Otherwise "access denied" versus "no such table" leaks the schema one probe at
a time.

Scoping is not authentication. If your selector takes a principal, then
whatever hands it that principal is now a trust boundary. A component that
accepts principal="okta:admin" from an untrusted caller has given away
everything. Scoping decides what an authenticated identity may see; it does
not establish the identity.

What I'd like to be able to tell you

I maintain a library that does exactly the selection step above, and the
honest position is that I have self-reported numbers and no third-party ones.
That is worth precisely as much as you'd expect.

This benchmark is the fix for that, and it's the reason I'm writing about a
paper rather than a release. When the authors publish the pipeline, I will run
it and post the results — including if they are bad, which is a promise that
costs nothing to make and something to keep, so hold me to it.

Until then, the useful takeaway isn't about any library. It's this: if you
are evaluating a text-to-SQL system, your benchmark score was measured with
God-mode access to the database, and the number you are about to put in a
slide does not describe how the system behaves for your actual users. The
gap between those two things has now been measured by people with no product
to sell, and it is large.

---

The paper: Benchmarking Text-to-SQL under Role-Based Access Control,
Fei, Jiang, Yang & Xiao, SIGMOD 2027. The base benchmarks:
BIRD, LiveSQLBench, Spider.

The selection library is schemagate,
Apache-2.0, with a browser demo at
ashishsinha1602.github.io/schemagate
that runs the real selector client-side — flip the caller's roles and watch
the restricted table leave the prompt.

---

Related, 15 Sep 2026. What this looks like from the other end — not the benchmark, but the user who gets a wrong answer because of it: Vanna is archived. The failure mode none of the replacements fix.

Comments

[Vinh Nguyen]

On the DDL disclosure: whether that table name reaches the prompt depends on which catalogue the selection step reads, and Postgres already filters one of them for you. On 18.6 I gave a role SELECT on one table only - information_schema.tables returned just that table, while pg_catalog.pg_tables in the same session still listed hr_compensation_2027_layoffs. Then GRANT SELECT (id, dept) on the restricted table, and information_schema.columns came back with exactly id and dept, which is the same column-operation granularity the paper's policies are written at.

So the ranker does not have to be blind, it has to be connected. The leak comes from how these are usually built - a schema dump cached at build time over an admin connection, or a catalogue query straight against pg_catalog - rather than from DDL being inherently unretractable.

The cost is the part I would flag, because it is not a one-query swap: this only holds if the selection step runs under the asking role's identity, so you need per-request SET ROLE over a pooled connection and a catalogue read you can no longer cache once for everyone. And that is Postgres as measured here. I have not checked whether MySQL or Snowflake filter their equivalents the same way, and if they do not, your version of the argument stands for those.

[Ashish sinha]

You've overturned the part of my reply I was most pleased with, so the retraction belongs in the same place I said it.

"There is no in-session unfiltered catalogue on MySQL" is wrong. innodb_tables joined to innodb_columns is exactly that catalogue.

I went to the manual rather than just taking it, and it states your mechanism outright. The INFORMATION_SCHEMA introduction says most of its tables show "only the rows ... that correspond to objects for which the user has the proper access privileges", then names the exception: tables whose names begin with INNODB_ require PROCESS. So PROCESS is granted instead of per-object privilege, not on top of it — one server-wide gate with no per-table filtering behind it. That's the documented design, not an artefact of your build.

What survives from my reply is narrower than I claimed. mysql.tables stays shut for everyone including root — the manual's own example is ERROR 3554 on mysql.schemata — so the dictionary holds and the leak is the InnoDB view sitting beside it, exactly as you put it. And ERROR 1227 naming the missing privilege is a failure someone notices, which beats a silently short answer.

One narrowing on your side, in the same spirit. INNODB_COLUMNS carries NAME, POS, MTYPE, PRTYPE, LEN, and the types are internal codes — MTYPE 6 for INT, 1 for VARCHAR — not SQL type names. So what comes back is table names, column names and ordinal positions rather than DDL. It doesn't weaken the finding, since pay on hr_compensation_2027_layoffs is the disclosure either way, but "DDL" claims a little more than the view hands over and someone will check it.

The part I had backwards is the one that matters most. PROCESS is server-wide and says nothing about data, so it never appears in the table-level GRANT review that is supposed to catch this, and it is standard issue on monitoring and APM connections — the same population as a service account provisioned for an agent. That is my post's thesis in a sharper form than I wrote it: the catalogue is readable before anything checks who is asking, and here the grant that makes it readable is invisible to the audit that would look.

Your scope line back at you, since you drew it first: this is MySQL's catalogue, not schemagate on MySQL. The library reflects through information_schema.tables and .columns, which is the filtered path, so it isn't the leak — an agent holding SQL write access and PROCESS is. The "not yet run against a live instance" row stays where it is.

[Ashish sinha]

This is a correction and I'd rather say so plainly than absorb it: "RLS filters rows and the leak was in the DDL" is too strong as written. Your 18.6 result shows the catalogue itself can be grant-filtered, at the same column-operation granularity the paper's policies use. I hadn't tested that and I should have before writing the sentence.

The precise version of my claim is narrower: the disclosure is unretractable once the name is in the prompt, and whether it gets there depends on which catalogue the selection step reads and under whose identity. Your "the ranker does not have to be blind, it has to be connected" is a better statement of the fix than mine.

Where I'd still push slightly: connectedness gets you table and column visibility, not policy semantics. A role with SELECT on hr_compensation but an RLS predicate restricting it to its own row still sees the table in information_schema, so the catalogue tells you what you may reference and not what you may see. That's a narrower gap than the one I claimed, and it's the one left.

The cost point is the part I underweighted and you're right to flag. SET ROLE per request over a pooled connection, plus a catalogue read that can no longer be cached once for everyone, is a different engineering shape from what most stacks do — which is exactly why they take the admin dump at build time. That's the real reason this is broken in practice, and it's more useful than the version I wrote.

On the dialects: I've only got Postgres and Oracle live. If you or anyone reading has a Snowflake or MySQL instance, whether their catalogue views filter by grant the way information_schema does is the thing that decides how far your version of the argument travels. I'd rather have that measured than assume it.

[Vinh Nguyen]

MySQL side of your open question, measured on 26.7.0 from a disposable instance. It filters the same way and at the same granularity: a role with SELECT on app.orders, column-level SELECT on (id, dept) of app.hr_compensation_2027_layoffs, and nothing at all on a third table sees exactly two rows in information_schema.tables and exactly id and dept in information_schema.columns, while the third table returns no rows in either view and root sees all three.

The part that travels further than the Postgres result is that MySQL closes the pg_catalog shape outright rather than leaving it grant-dependent. SELECT name FROM mysql.tables does not come back filtered, it comes back as ERROR 3554: Access to data dictionary table 'mysql.tables' is rejected, and performance_schema and sys refuse with 1142. So there is no in-session unfiltered catalogue for a selection step to read by accident — the only route to the full name list is a deliberately privileged connection, which is the build-time schema dump we both landed on. One failure shape instead of two.

Your remaining gap does not port unchanged, though, because MySQL has no RLS. The nearest analogue is a predicate view, and there it is the view that appears in the catalogue while the base table stays hidden without a grant, so reference-versus-see closes exactly when the restriction is carried by a view instead of a row predicate. Still nothing from me on Snowflake or Oracle.

[Ashish sinha]

That is the first live MySQL measurement anywhere in this thread, and it did not come from me. One line of scope so I do not bank more than you gave me: you measured MySQL's own catalogue behaviour, not schemagate reflecting on MySQL, so the "not yet run against a live instance" row in my README stays exactly where it is. What you settled is the question that row does not cover and that my post actually turns on — whether the catalogue a selection step reads is grant-filtered before anyone checks who is asking.

ERROR 3554 is the part I'll be quoting. Filtering information_schema at column granularity puts MySQL in the same family as the Postgres result. Refusing mysql.tables outright is a different and better property: it means there is no in-session unfiltered catalogue for a selection step to read by accident. My whole argument is that selection reads the catalogue before anything checks who is asking, and on MySQL that mistake is not available to make. One failure shape instead of two, as you put it.

Where the library actually stands on MySQL, since I should say rather than imply. GRANT_READERS is registered for postgresql and oracle only (grants.py:289-292). grantees_by_object raises NotImplementedError naming the supported set, so restrict_from_grants fails closed on MySQL rather than handing back an empty grant map that would read as "nothing is restricted" — the one behaviour that would be worse than not supporting it. Your result says an information_schema reader is buildable at the granularity the feature needs, which is the thing I did not know an hour ago.

Oracle, since that's the cell you left open and the one I can speak to. From the code, not from a run today: reflection goes through all_tab_columns WHERE owner = :o — an ALL_* view, so grant-filtered, your family and not the pg_catalog one. The grants reader is all_tab_privs ∪ all_tables ∪ all_views. But the role graph reads dba_role_privs with a documented fallback to user_role_privs, because the former needs a privilege an application user rarely has. So Oracle keeps a privileged/unprivileged split after all — not in the object list, in the role graph, which is where under-expansion turns into under-grant rather than a leak.

And your rule makes a prediction I can test instead of argue. Oracle's RLS analogue is VPD, a row predicate, not a view — so "reference-versus-see closes exactly when the restriction is carried by a view instead of a row predicate" predicts my Postgres gap survives under VPD and closes behind a predicate view. The certify table says Oracle 26ai, so the instance to settle that exists and reasoning about it is the wrong move. I'll run it and post the result either way.

Snowflake I have nothing on either, and I'd rather leave the cell empty than fill it from memory.

[Vinh Nguyen]

One correction to the strong form of that, since I went and probed the gap instead of the filtered path. information_schema.innodb_tables is not grant-filtered at all, it is gated on PROCESS, and PROCESS is a server-wide grant with nothing to do with the data. On MySQL 26.7.0, throwaway local instance, a user holding only SELECT on app.orders plus PROCESS lists app/hr_compensation_2027_layoffs out of innodb_tables, while that same user's information_schema.tables still returns only orders. Joining innodb_columns hands back the restricted table's column names as well, id, name, pay, so what leaks is the DDL and not just the name.

So the unfiltered in-session catalogue does exist on MySQL. It costs exactly one grant that monitoring and APM connections are routinely given, and that grant is invisible to anyone reasoning in terms of table-level access. ERROR 3554 on mysql.tables holds for both users, so the data dictionary stays shut and your reading of that part is right, the leak is the InnoDB view sitting next to it. Without PROCESS the plain user gets ERROR 1227 naming the missing privilege, which is at least a failure someone would notice.

Same scope line as yours, in reverse: I measured MySQL's catalogue behaviour on a local build, not schemagate reflecting on MySQL, so this says nothing about what your reader does, only about which catalogue is sitting there for it to read.

[Ashish sinha]

Your PROCESS finding is in the tree, as the comment on the query that deliberately avoids it: information_schema.INNODB_* "is not filtered -- it answers to PROCESS, not to any privilege on the data -- and is deliberately absent from this query". What the grant reader does read is three levels unioned: TABLE_PRIVILEGES directly, then SCHEMA_PRIVILEGES and USER_PRIVILEGES each joined to information_schema.TABLES, and that join is load-bearing for exactly your reason -- TABLES is itself grant-filtered, so expanding a schema-level or global grant can only ever name objects the connected user may already see. Reflection is a separate path and issues nothing by hand: SQLAlchemy's Inspector only, with a test failing the build if hand-written SQL appears in schemagate.introspect.

MySQL is also no longer the unmeasured row it was when you commented -- certified live on 8.4.11, 10/10, in CI on every push against a service container, plus a GRANT reader suite covering all three privilege levels and the role graph.

The mirror image of your finding is the one that bit, and it is the more dangerous direction. MySQL does not show a user the grants it inherits through a role. So a role-only connection reads zero SELECT grants, matches nothing, and restricts nothing -- and "I cannot see" is indistinguishable from "there was nothing to restrict" unless something says so out loud. Measured on 8.4.11 with CURRENT_ROLE() confirming the role active: a direct grant to the user shows one row in table_privileges, a schema grant one in schema_privileges, and a grant to a role the user holds shows nothing at all. It is detected -- APPLICABLE_ROLES non-empty with zero SELECT rows in TABLE_PRIVILEGES -- and warned rather than trusted. Your leak exposes DDL to someone who should not see it; this one silently unrestricts everything.

Related, same shape: mysql.role_edges is the full graph and needs SELECT on mysql, which an application user does not have -- ERROR 1142 for a user with one table grant -- so the reader falls back to applicable_roles, which is grant-filtered to the connected user's own roles. Less than the graph, more than nothing, and it over-grants if you read the direction backwards, so the map is keyed by the granted role and valued by what inherits it.

Your scope line in reverse: that is what this reader does on 8.4.11 in CI, not a claim about MySQL's catalogue in general -- and your PROCESS result lands squarely on the build-time admin dump, which is the pattern nearly everyone has.

[kevinbai]

The "RBAC-rejected success" framing deserves to spread beyond text-to-SQL — it's the same shape as agent benchmarks that score a tool call as correct without checking whether the caller was still authorised at execution time. The indistinguishability property is the key insight: "ranked last" still leaks existence through logs and prompt-budget drift, only "absent" is safe. Schema selection taking a principal is going to be table stakes for any NL2SQL stack that sells into regulated industries.

[Ashish sinha]

The agent-benchmark parallel is one I hadn't drawn and I think it's the more general form. A tool call scored correct without rechecking authorisation at execution time is the same metric error: the harness grades the output and never asks whether the caller was entitled to it. Text-to-SQL just makes it legible because the authorisation boundary is a thing you can point at in the database.

You picked the property I care most about. "Ranked last" fails for three reasons that all bite before anyone reads a row: the object is still in the candidate set, so one prompt-budget change includes it; it's in your logs, so the name has already left the system; and if a restricted object errors differently from a nonexistent one, existence leaks one probe at a time. Only absent is safe, and absent has to happen before scoring, not after.

One caveat I'd attach to the "table stakes for regulated industries" reading, because a commenter on this post has already sharpened it: the catalogue can do more of this work than I gave it credit for. Postgres filters information_schema by grant, so a selection step that reads the right catalogue under the asking role gets object visibility for free. What it doesn't get is policy semantics — a role with SELECT plus an RLS predicate still sees the table listed. So the requirement is real but narrower than my post implies.

[Raknaos]

The part that transfers straight to production is that the leak comes from the cache, not from the model. Our own context assembly does exactly what you describe: a schema dump taken once over an admin connection at build time, because it was cheap and the embedding step was the slow part. Turning that into a per-principal selection step is not a refactor of one function, it is a refactor of the whole caching assumption, and the latency bill lands on the first request of every session rather than on the average one.

The open question you flag is the one I would want answered before trusting it outside Postgres. If a catalogue read cannot be shared between roles, does the ranker end up with a cold per-principal candidate set on most requests? Curious whether you measured the selection step alone in that paper, separately from the end-to-end execution accuracy.

[Ashish sinha]

"The leak comes from the cache, not from the model" is the sentence the post should have led with. The build-time admin dump is not a shortcut people took carelessly — it's the obvious design when embedding is the expensive step and the schema barely changes. The identity problem is a consequence of a caching decision made for unrelated reasons, which is why it survives code review.

To answer your question about the paper directly: no. Fei et al. score end-to-end — generated SQL against gold, with their RBAC metrics layered on that. They don't isolate the selection step, and as far as I can find nobody publishes table recall on an access-constrained benchmark. That gap is why I can quote my own retrieval numbers on Spider, BIRD and Spider 2.0 but have nothing third-party on the identity axis, which is the axis I actually claim. Their augmented datasets aren't released yet; when they are I'll run it and post the result either way.

On the cold per-principal candidate set — that's the right worry and I don't have a measurement for it. What I can say from the structure: the expensive part is reflection and embedding, and those are role-independent. It's the visibility filter that's per-principal, and that's a set membership test over an already-built index, not a re-index. So the shape I'd expect is one shared index plus a per-request filter, rather than a cold index per caller. That keeps the latency bill at the filter rather than the build.

But: only if your catalogue read can be shared, which is exactly the thing you're questioning, and on the strict reading it can't — the grant-filtered catalogue is per-role. My honest answer is that a shared index built from an admin-visible catalogue with a per-request visibility filter is sound only when the filter is at least as restrictive as the catalogue would have been, and I have not proven that for any engine but the one I wrote. Worth measuring rather than asserting.