TL;DR — A dashboard is a question that humans reviewed, argued over, and corrected. A text-to-SQL agent produces unreviewed questions at scale, and its typical failure is a valid query for the wrong interpretation. Fix this with answer receipts, divergence-triggered clarification, and a path that promotes frequent agent questions into reviewed artifacts.
Most conversations about text-to-SQL agents are about SQL generation: schema linking, join paths, dialect quirks. Those problems are real, and they are also the ones that get better on their own as models improve.
The problem that doesn't fix itself is this: a dashboard is a reviewed question, and an agent answer is an unreviewed one. Analytics engineers have been relying on that difference for years without naming it.
What a Dashboard Actually Was
We talk about dashboards as visualization. Their real function was social validation of a query.
Someone chose the grain. Someone decided what "active user" means, which timezone cuts the day, whether refunds net out, which test accounts get excluded. Then the chart sat in front of dozens of people who knew roughly what the number should be. A wrong figure got noticed because a finance lead said "that can't be right," and the query was fixed. Over months, the artifact converged.
That loop was never in the architecture diagram. It was implicit error correction performed by humans glancing at a shared artifact repeatedly.
A conversational agent deletes the loop. The question is asked once, answered once, and consumed by one person who has no strong prior about the answer, which is often exactly why they asked.
The Failure Is Interpretation, Not Syntax
Benchmarks for text-to-SQL mostly score whether the generated query returns the same result as a reference query. That is a reasonable proxy for translation. It is a poor proxy for answering a business question, because the reference query encodes one interpretation chosen by whoever wrote the benchmark.
In production, the dangerous output isn't a query that fails. A failing query is loud. The dangerous output is a query that runs cleanly, returns a plausible number, and answers a slightly different question than the one the user meant. Consider "revenue last quarter":
Calendar quarter or fiscal quarter?
Gross, net of refunds, or recognized?
Booked date or paid date?
Which timezone defines the boundary?
Does it include internal or test accounts?
Each choice produces a defensible query. The results differ, sometimes by a little, sometimes a lot. The agent picks one and returns a number with the confidence of a calculator.
A semantic layer helps here, but only partly. It pins down what a metric means once you've selected it. It does not pin down the time window, the population, or which of three similar metrics the user was reaching for. The ambiguity migrates upward into question interpretation, and that part of the system has no owner.
Receipts: Make the Interpretation a Return Value
The first fix is cheap and architectural. An agent should never return a bare number. It should return an answer receipt, a structured record of the choices it made:
The resolved metric and the definition it bound to, by identifier rather than prose.
The grain, filters, time range, and timezone, stated explicitly, including defaults the user never specified.
The executed SQL.
Input freshness and the row counts at key stages, so an empty join is visible.
Any assumptions the agent made because the question was silent.
The point is not to make users read SQL. Most won't. The point is that the receipt is machine-checkable and human-skimmable. A finance lead can scan "fiscal quarter, net of refunds, UTC" in two seconds and catch the mismatch, which is exactly the glance that a dashboard used to provoke.
Defaults are the critical part. Every silently applied default is a decision the user didn't make. Surfacing the defaults turns hidden interpretation into reviewable interpretation.
Ask When Interpretations Diverge, Not When the Model Feels Unsure
The obvious next step is "have the agent ask clarifying questions." Done naively, it's miserable. The agent asks about everything, users learn to click through, and you've built a worse form UI.
Model self-reported uncertainty is a weak trigger. A better one is observable: generate several plausible interpretations, execute them, and compare the results.
If the candidate interpretations all return roughly the same number, the ambiguity is harmless and the agent should just answer, with a receipt. If they diverge beyond a tolerance, the ambiguity is material, and the agent should surface the fork: "Net of refunds this is one figure; gross it is another. Which do you want?"
This reframes clarification as a cost-benefit decision grounded in outcomes instead of vibes. It also has an honest cost, since you run multiple queries per question. That cost is bounded and predictable, and cheap compared to a wrong number making it into a planning meeting. You can limit it by running the alternates only on questions whose terms resolve to more than one candidate definition.
Close the Loop: Promote Questions Into Artifacts
Receipts and divergence checks make individual answers safer. They don't rebuild the thing dashboards gave us, which was convergence over time.
To get that back, log every receipt and cluster the questions. The frequent clusters are your real demand signal, and they should graduate:
An agent answers a question with a receipt.
The same intent shows up repeatedly across users.
An analytics engineer reviews the interpretation, fixes definitions, and certifies it as a named query or metric.
The agent routes future matches to the certified artifact instead of regenerating SQL.
The dashboard becomes a cache of vetted questions, and the agent becomes the long tail around it. Certified paths get the reliability of reviewed work. Novel questions get flexibility, clearly labeled as unreviewed.
That label matters. Users should be able to tell "this came from a certified definition" from "I improvised this just now." Trust calibrates correctly only if the system exposes the difference.
What This Changes About the Job
If agents write the SQL, the analytics engineer's value doesn't vanish. It moves from writing queries to curating the question space: deciding which interpretations are canonical, retiring definitions that conflict, and reading divergence logs to find where the business itself is ambiguous.
Those logs are surprisingly revealing. When two interpretations of "active customer" keep producing different numbers, the agent didn't fail. It found a real disagreement in your organization that dashboards had been papering over by letting each team keep its own chart.
The takeaway for anyone shipping these systems: stop measuring only whether the SQL is right. Measure whether the interpretation was visible, whether material ambiguity was surfaced, and how often an unreviewed question earns its way into a reviewed one. An agent that is right 90% of the time and hides which 10% is worse than one that is right 80% of the time and tells you where to look.
Top comments (0)