Every demo shows one question. Analysis happens at question six.
Conversational analytics is judged on the first query and lives or dies on the fifth. What has to hold across turns — and what quietly stops holding.
Watch any demo in this category and you'll see the same thing: one question, one query, one chart, applause. It's a fair demonstration of a solved problem. It is not a demonstration of analysis.
Real analytical work is a loop. You run something, notice an anomaly, narrow it, compare it against a baseline, notice something else, chase that. The first query is the cheapest part of the session; everything interesting happens in the follow-ups. Which means the question worth asking of any conversational tool isn't "can it write SQL" — it's what survives from one turn to the next.
A real session, five turns deep
Turn 1 — "Show revenue by month this year."
SELECT date_trunc('month', completed_at) AS month, sum(net_amount) AS revenueFROM analytics.v_ordersWHERE completed_at >= date_trunc('year', now())GROUP BY 1 ORDER BY 1;
Turn 2 — "Now just enterprise customers."
SELECT date_trunc('month', o.completed_at) AS month, sum(o.net_amount) AS revenueFROM analytics.v_orders oJOIN analytics.v_accounts a ON a.id = o.account_idWHERE o.completed_at >= date_trunc('year', now()) -- carried from turn 1 AND a.plan_tier = 'enterprise'GROUP BY 1 ORDER BY 1;
What had to carry: the date range from turn one. Adding a filter must not quietly reset the window — and a join was introduced, which is where grain problems start.
Turn 3 — "Compare that with last year."
SELECT date_part('month', o.completed_at) AS m, date_part('year', o.completed_at) AS yr, sum(o.net_amount) AS revenueFROM analytics.v_orders oJOIN analytics.v_accounts a ON a.id = o.account_idWHERE o.completed_at >= date_trunc('year', now()) - interval '1 year' AND a.plan_tier = 'enterprise' -- must surviveGROUP BY 1, 2 ORDER BY 1, 2;
Where it usually breaks: "that" has to mean enterprise revenue by month, not revenue by month. Weak tools silently drop the enterprise filter here and hand back a comparison of the wrong population — with no error and a perfectly reasonable-looking chart.
Turn 4 — "Which months were down year over year?"
-- operates on the previous result, not a fresh questionSELECT m, cur.revenue, prior.revenue, cur.revenue - prior.revenue AS deltaFROM (/* turn 3, current year */) curJOIN (/* turn 3, prior year */) prior USING (m)WHERE cur.revenue < prior.revenue;
Where it usually breaks: this turn doesn't restate any filter, any metric, or any population. Everything it needs is inherited. A tool that treats each message as a fresh prompt cannot answer it correctly — and will answer it anyway.
Turn 5 — "What's driving March?"
SELECT a.name, sum(o.net_amount) AS revenue, count(*) AS ordersFROM analytics.v_orders oJOIN analytics.v_accounts a ON a.id = o.account_idWHERE date_trunc('month', o.completed_at) = '2026-03-01' AND a.plan_tier = 'enterprise' -- still carriedGROUP BY 1 ORDER BY 2 DESC LIMIT 20;
What had to carry: the grain changes completely — from a time series to a per-account breakdown — while the population filter stays.
Five turns, and only the first one was a self-contained question. The other four are almost entirely inherited context. That's not an edge case; that's what analysis is.
The research says the same thing
This isn't only an observation from watching demos. The field has a dedicated benchmark for it: CoSQL, built at Yale and published at EMNLP 2019, is the conversational counterpart to the single-turn Spider benchmark — 3,000 dialogues across 200 databases, including questions that aren't answerable and ones the system should ask about rather than answer. Its sibling, SParC, covers sequential question-answering over the same ground.
The interesting part is how CoSQL scores. It reports two numbers: question match, how often an individual turn is right, and interaction match, how often the entire conversation is right. They are reported separately because they diverge sharply — a leaderboard entry scoring 57.8% per question managed 28.2% across complete interactions.
Those particular figures are from 2022 models and current ones do considerably better. The structure of the problem hasn't moved, though, and the benchmark existing at all is the point: conversational text-to-SQL was recognised early as a different problem from single-turn, because performing well at one predicts much less than you'd hope about the other. A demo is a question-match test. Your Tuesday afternoon is an interaction-match test.
What "remembering the conversation" has to mean
There's a weak version of conversational memory and a strong one, and they look identical for about three turns.
The weak version passes the chat transcript back to the model as text and asks it to work out what you meant. It handles "now filter for enterprise" fine. It starts failing at "compare that with last year," because resolving that requires knowing the structure of the previous query, not just the words that preceded it.
The strong version carries the query itself as state — the active filters, the population, the metric, the grain — and treats each turn as a transformation of that state rather than a new request with extra words attached. Then "compare with last year" is a well-defined operation on a known object, not an inference problem.
Which one you're using is invisible in a demo and obvious by turn five.
Four ways multi-turn goes wrong
Filters silently drop. The most common. You narrowed to enterprise three turns ago; a reframing turn quietly returns you to the whole population. Nothing announces it, the chart still looks sensible, and every conclusion after that point is about the wrong group.
Definitions drift between turns. Turn one used net revenue. Turn four, phrased differently, uses gross. Two numbers in the same session that aren't comparable, and no indication that the basis changed.
Pronouns resolve to the wrong thing. "Those customers," "that number," "the same period" — each one is a reference that has to bind to something specific. When the binding is guessed rather than tracked, it's usually guessed plausibly, which is the problem.
Nobody can say what's currently applied. By turn six, what's actually in the WHERE clause? If answering that means reading back through the whole conversation, the accumulated state has become unverifiable — and someone is about to screenshot it.
All four share one trait with the failures catalogued in seven ways AI SQL goes wrong: the query still runs. There is no error for inheriting the wrong context, which is why the fix isn't better error handling — it's making the current state visible.
What good looks like
A tool that handles iteration well does three unglamorous things. It shows the active state — the filters, the population, the date range currently applied — somewhere you can see without scrolling. It shows the SQL every turn, not just the first, so a drift in turn four is visible when it happens rather than at reconciliation. And it lets you branch or reset deliberately, because half of analysis is backing out of a direction that didn't pan out.
None of those are AI features. They're interface decisions, and they're what separates a tool you can do real work in from one that demos beautifully.
How to test it in ten minutes
Run exactly the session above against any tool you're evaluating, on your own data. Ask the five questions in order, in that phrasing, and read the SQL at every turn.
Then do the one thing nobody does: go back to turn three and ask it again, differently. "How does that compare to the same months last year?" If the answer changes, the tool is inferring context rather than tracking it — and you've learned more in ten minutes than a feature comparison would tell you in a week.
That test belongs alongside the rest of them in our seven-test harness.
Built for turn five. DBx Studio carries the query as state across a conversation, shows what's currently applied, and keeps the generated SQL visible on every turn — so the sixth answer is as checkable as the first.
Query it. Analyze it. Visualize it. — all with DBx.
Top comments (0)