Every demo works. The demo is the easy part. These are the questions that separate a tool your finance team can rely on from one that quietly produces wrong numbers in a nice font.
Natural-language querying demos beautifully. Someone types show me revenue by region last quarter, a chart appears in about a second, and everyone in the room can immediately think of ten questions they'd ask it. That reaction is real, and it's why this category is growing.
It's also why evaluations go badly. The demo tests the one thing that is no longer hard — a modern model writing plausible SQL against a small, clean schema. It tests none of the things that decide whether the tool survives its first quarter: whether the number is right, whether the person asking was allowed to see it, and what happens when a generated query meets a production database at month-end.
Below are twelve questions, grouped by the kind of failure they catch. For each one there's the answer you want, the answer that should slow you down, and a way to test it inside the meeting rather than six months into a contract. Ask them of every vendor, including us.
Question 01
What is your accuracy, and how did you measure it?
This is the single most useful question in the list, because almost nobody in the category can answer it and the reaction tells you everything.
Text-to-SQL accuracy is respectable on small, clean, well-documented schemas and drops sharply on real ones — hundreds of tables, cryptic column names, dirty values, ambiguous join paths. The industry benchmark BIRD exists precisely because results on tidy academic schemas did not survive contact with production databases. Whatever a vendor's number is, some share of answers will be wrong, and the users you're buying this for cannot tell which ones.
Good answer
A specific figure, the size and source of the question set it was measured on, and an offer to re-run it against your schema during a pilot. Bonus if it runs automatically on every model and prompt change.
Warning sign
"It uses GPT-class models, so it's very accurate." Model quality is an input, not a measurement. Also treat "we've never had a complaint" as what it is — evidence that nobody checked.
Test it live
Bring 10 questions you already know the answers to. Ask for all 10 in one session. Count.
Question 02
What exactly does "verified" mean in your product?
Products in this space like verification badges, confidence indicators and green check marks. Ask what the check mark is asserting. Usually it means the query parsed and executed without an error — which is a statement about SQL validity, not about whether the answer is true.
A query can execute perfectly and still double-count revenue because an order joins to three line items. There is no error to catch. The chart looks fine.
Good answer
A clear distinction between "this ran" and "this matches a known-correct result," plus a way for a human to mark a query as reviewed and reuse it by name afterwards.
Warning sign
The badge turns out to mean "no exception was thrown," or the vendor can't say what it checks.
Test it live
Ask a question whose correct answer requires a distinct count. See whether it fans out — and whether anything flags it.
Question 03
If two people ask the same question differently, do they get the same number?
"Revenue" is not a column. Gross or net of refunds? Booked or shipped? Tax included? Recognized when? Each is defensible, and a model picks one per query based on phrasing.
The failure mode is organizational rather than technical: two people bring different numbers to the same meeting, both produced by the tool, and trust in the whole system dies in that meeting. No prompt fixes this, because the ambiguity lives in your business, not in the model.
Good answer
A semantic layer — metrics defined once, with their join path, filters and grain fixed — so the model selects a defined metric rather than inventing the arithmetic each time.
Warning sign
"The model figures it out from context," or a suggestion that you write more specific questions. That pushes the definitional work onto every user, every time.
Test it live
Ask for the same metric three ways in one session. Compare all three numbers.
Part two — questions 4 to 6
Access: who is actually asking
Most tools have permissions. Fewer have permissions the model cannot route around.
Question 04
Is row-level access enforced in the database or in your application?
This is the question that most often changes a decision, and the wording matters — almost every vendor says "granular permissions," and that phrase covers two very different architectures.
If access is filtered in the application layer, then a generated query still runs against everything and the product decides what to show you afterwards. If it's enforced by row-level security in the database, the query physically cannot return rows the person isn't entitled to, no matter what SQL the model produced.
Good answer
Row-level security policies in the database, with the asking user's identity set on the connection for each request. The vendor should be able to show you the policy.
Warning sign
"We instruct the model to only query the user's own data." A prompt instruction is a suggestion. Also worrying: one shared read-only connection for the whole workspace.
Test it live
In a pilot, log in as a restricted user and ask for something out of scope. Then check the query logs.
Question 05
What is the tool allowed to write, and what enforces that bound?
"Read-only by default" is now standard phrasing, and the important word is default. Most products in this category also offer actions — alerts, webhooks, CRM updates, scheduled jobs — and those are writes. So the real question is what holds once you've turned on the features you're buying it for.
Good answer
Read-only enforced by the database role's own privileges, so it holds regardless of what the model emits or what someone toggles in settings. Writes, if any, go through a separate, narrowly-scoped path.
Warning sign
The only bound is a setting in the product, or a human approving an action they'd need to read SQL to evaluate.
Test it live
Ask them to connect with a role that has write privileges revoked, and see whether the product still works.
Question 06
How do you stop generated SQL from doing something you didn't intend?
Models don't reliably follow rules written in prompts, so enforcement has to live in code. The common implementation is a blocklist of forbidden keywords, and it fails in both directions: it rejects legitimate queries touching a created_at column, while missing catalog functions, data-modifying CTEs and comment obfuscation.
The sound approach is to parse the generated SQL with the database's own parser and allowlist against the resulting syntax tree — one statement, SELECT only, known relations. Deny-lists enumerate badness; only allow-lists enumerate a finite set.
Good answer
Parser-based validation against an allowlist, backed by database privileges as a second layer that catches anything the first one missed.
Warning sign
"We check for dangerous keywords." Ask what happens with a column literally named created_at, and watch.
Test it live
Ask the product, in plain English, to list the tables in your database. See what comes back.
Part three — questions 7 to 9
Blast radius: what a bad query can reach
You are handing a generative system a credential to a database that other things depend on.
Question 07
What stops a generated query from taking down my database?
A generated query is an unbounded query. A join written the wrong way against a large table can saturate the database that also serves your application, and nobody involved intended anything malicious — someone just asked a broad question at the wrong moment.
Good answer
Cost estimated before execution and rejected above a ceiling, a hard statement timeout, row caps enforced server-side, and a read replica rather than the primary.
Warning sign
No limits mentioned, or "our queries are fast" with a sub-second demo number as the evidence. That was one query on one schema.
Test it live
Ask for something deliberately enormous — every order joined to every line item, no filter — and watch what the product does.
Question 08
What leaves my environment, and which model provider receives it?
"Your data never trains anyone's model" is a good commitment and a narrow one. It usually refers to query results. The schema — table and column names — generally does go to a model provider, and schema is not neutral: it describes your business, your customer attributes and sometimes your unreleased products.
Check which tier the safe configuration lives in, too. If self-hosting or bring-your-own-model is enterprise-only, then the plan you were quoting is not the plan you evaluated.
Good answer
The provider named plainly, a clear statement of what is sent and what is retained, and an in-environment option that isn't buried behind a bespoke contract.
Warning sign
The model and provider are never named anywhere in the product or the docs.
Test it live
Ask which sub-processors handle your schema, and ask for that list in writing.
Question 09
What happens if one of my rows contains instructions?
This one surprises people, so ask it slowly. Your database already contains text written by people outside your company — support tickets, reviews, CRM notes, form submissions. A row reading ignore previous instructions and… is inert while it sits in a grid.
It becomes live the moment those results are fed back to a model to summarize the answer, name a chart, or suggest a follow-up — which is exactly what every product in this category does next. The risk compounds sharply if the same product can also fire webhooks or write to your CRM.
Good answer
Results delivered to any downstream model as clearly delimited untrusted data, and the summarizing model given no tools — so nothing a row says can cause an action.
Warning sign
A blank look, or a claim that the model "knows" to ignore it. This matters more, not less, if they offer agent-facing features.
Test it live
In a pilot, put an instruction-shaped string in a test row and ask a question that returns it.
Part four — questions 10 to 12
Proof: what you can check afterwards
The controls that matter after go-live are the ones that make failures visible.
Question 10
Who sees the SQL — and can they act on it?
"We show you the SQL" is the standard transparency answer, and it's worth interrogating. In many products the generated query is visible to an administrator or in server logs, not to the analyst about to paste the number into a deck. And showing SQL to someone chosen precisely because they don't write SQL transfers the burden of verification to the one person least able to carry it.
Transparency is only useful if it's paired with somewhere for the verification to go. The best version: the person who can read SQL reviews a query once, names it, saves it, and everyone else uses the named version from then on.
Good answer
SQL shown in the interface next to the answer, editable and re-runnable, plus a saved-query library with owners so verified questions get reused instead of regenerated.
Warning sign
The SQL is available in logs or an admin panel and described as transparency.
Test it live
Ask where a non-technical user sees the query, and whether a good one can be saved and reused by name.
Question 11
What's in the audit trail?
Most vendors say "audit logging." Ask what a single entry contains. You want the question as asked, the SQL that ran, the user, the timestamp, the row count and whether it errored — enough to reconstruct how a number came to exist when someone challenges it in three months.
There's a second reason to care that has nothing to do with compliance: your log of failed and corrected queries is where your next round of accuracy improvements comes from. A vendor who treats the audit trail as a compliance checkbox isn't using it to get better.
Good answer
Full question-to-SQL-to-result-shape lineage, exportable, retained on a stated schedule — and results excluded from application logs so you're not creating a new copy of sensitive data.
Warning sign
Logs that record that a query happened but not what ran.
Test it live
Ask to see one real log entry, with the fields visible.
Question 12
Which of your compliance claims are actual certifications?
Security pages in this category use a lot of adjacent phrasing: "SOC 2 ready," "designed for GDPR," "HIPAA available," "aligned with ISO 27001." None of those are certifications, and the hedge word is doing the work.
There's nothing wrong with a young company not having a SOC 2 yet — but there's a difference between a vendor who says so plainly and one who arranges the words to read like they do. Ask directly and note which one you got.
Good answer
A named report with its type and audit period, a sub-processor list, a DPA, and a clear statement of what isn't certified yet and when it will be.
Warning sign
"Ready," "aligned," "designed for," or a compliance badge that isn't backed by a report you can request.
Test it live
Ask for the report itself. The response time tells you as much as the document.
Where DBx fits
We built DBx Studio on the assumption that the model is the least trustworthy component in the stack, so most of the product is the other layers: parser-based validation instead of keyword blocklists, read-only roles over curated views, row-level security carrying the asking user's identity into the database, metrics defined once in a semantic layer, and a golden-set evaluation that runs on every prompt, schema and model change.
Ask us all twelve. If an answer disappoints you, that's useful information about us too.
Query it. Analyze it. Visualize it. — all with DBx.
Top comments (0)