TL;DR — Natural-language BI agents let anyone ask questions of a warehouse without writing SQL, but the analytics layer underneath still needs the same rigor as any production system: versioned metric definitions, tested transformations, and reviewable logic. The risk isn't that agents write bad SQL — it's that they write plausible SQL against an unowned semantic layer, and nobody notices until a board deck is wrong. Analytics engineering's job just got bigger, not smaller.
Every BI vendor now ships a chat box. Ask a question in plain English, get a chart back. The demo is always the same: someone types "what were sales by region last quarter" and a bar chart appears in four seconds. It's genuinely impressive. It's also the most dangerous four seconds in your data stack, because nobody in that demo asks the question that matters: which "sales" did it use?
The Agent Isn't the Risk. The Ambiguity It Inherits Is
Analytics engineers have spent the last several years building dbt models, defining metrics once, and fighting the eternal war against spreadsheet-driven "shadow metrics" that don't match the warehouse. That war was mostly won by discipline: pull requests, tests, a single source of truth for what "active user" or "net revenue" means.
Agentic BI threatens to undo that discipline by making it invisible. An LLM sitting on top of your warehouse doesn't know that revenue excludes refunds in one schema and includes them in another. It doesn't know that "last quarter" is fiscal, not calendar, for finance but calendar for marketing. It will confidently generate a query anyway, because generating plausible SQL is exactly what these models are good at. The query will run. The chart will render. The number will be wrong, and it will look exactly as trustworthy as the right one.
This isn't a hypothetical about model quality improving fast enough to fix itself. Ambiguity in business definitions isn't a language problem — it's an organizational problem, and no amount of scaling makes an LLM know something that was never written down anywhere it can read.
The Semantic Layer Just Became Load-Bearing
Metric layers, semantic layers, whatever you call them — dbt's metrics, Cube, LookML, headless BI — used to be a nice-to-have that let you define a metric once and reuse it across five dashboards. In an agent-driven workflow, they stop being a convenience and become the actual interface between natural language and truth.
Here's the shift: when a human analyst writes SQL, their mental model of the business fills the gaps the schema doesn't cover. They know "sales" means net sales because they've been in the Tuesday revenue review for two years. An agent has no Tuesday revenue review. It has whatever is encoded in table names, column comments, and metric definitions — and nothing else. If your semantic layer is thin, the agent's fluency doesn't compensate for it. It hides it.
That means the real infrastructure investment for "AI-powered BI" isn't a better model or a slicker chat UI. It's an unglamorous, extremely necessary push to make every metric definition explicit, versioned, and machine-readable — the same rigor you'd expect from a config file that gets deployed to production, because that's exactly what it is now.
Treat Metric Definitions Like an API Contract
If an agent is going to generate queries against your metrics, those metrics need the properties every stable API needs:
A single canonical definition per metric, with no duplicate logic living in five dashboards under slightly different names.
Versioning, so when finance redefines "churned customer," old reports don't silently start recalculating history under the new rule.
Tests that catch drift — the same kind of assertion testing you already run on dbt models, extended to check that metric outputs match known-good values on a fixed dataset.
Ownership, meaning a specific team is accountable for what a metric means, the same way a specific team owns a production API's schema.
None of this is new advice for analytics engineering. What's new is the cost of skipping it. Before agents, a bad metric definition produced one bad dashboard that a sharp analyst might catch on review. Now a bad metric definition gets multiplied across every ad hoc question anyone asks the chat interface, forever, with no review step in between. The blast radius of an unowned metric just went from one report to unlimited reports.
Agents Are Good at Syntax, Not Judgment
It's worth being precise about what these systems are actually doing well. Text-to-SQL generation, when it's grounded in a real semantic layer rather than raw table introspection, is legitimately strong now. Given a clean metric definition and a clear question, an agent will write correct, efficient SQL more reliably than a mid-level analyst on a Friday afternoon.
What agents cannot do is exercise judgment about which metric is appropriate for which question, because that judgment was never data — it was context that lived in people's heads and Slack threads. "Should this report use gross or net revenue" is not a SQL problem. It's a business problem that has to be resolved and codified before the query gets generated, not inferred by the model at query time.
This is the same lesson the industry has learned repeatedly with structured output and RAG: giving a model the right shape of output, or the right grounding documents, doesn't automatically give it the right meaning. Syntax and retrieval are solved problems. Semantics is the permanent, unglamorous work, and it doesn't get automated away — it gets exposed.
What Analytics Engineers Should Actually Build Next
The practical implication is a reordering of priorities. Building another dashboard is now lower leverage than auditing and consolidating the metric layer that every future dashboard — human-built or agent-generated — will draw from. Concretely:
Inventory every place a core metric is defined and collapse it to one place, with the agent-facing semantic layer as the only allowed source.
Write regression tests for metrics the way you'd test a payments calculation, not the way you'd eyeball a chart.
Log every agent-generated query against the metric layer, and review the ones that touch ambiguous or recently changed definitions.
Make disambiguation explicit in the interface — when "revenue" has two valid meanings, the agent should ask, not guess.
None of this is exciting work. It won't make a good demo. But it's the difference between an agent that makes your analytics team faster and one that makes your analytics team the last line of defense against numbers that were wrong three clicks before anyone noticed. The chat box didn't remove the need for analytics engineering discipline — it just moved the failure mode from "bad dashboard" to "bad answer delivered with total confidence, at scale, to someone who trusted it."
Top comments (0)