Every AI startup I walk into has a graveyard of dashboards. There's the one someone built during the last incident, still pinned even though nobody remembers what it was for. There's the "overview" that tries to show everything and therefore shows nothing. There's a folder of near-duplicates — extraction quality, extraction quality (v2), extraction quality FINAL — each spawned by a question that the previous one couldn't quite answer.
And still, the day something breaks, the dashboard you actually need does not exist.
It's 2 a.m. Extraction quality on your document pipeline is down for one specific customer, on one specific document type, since roughly Tuesday. You have forty charts. None of them is that chart. So you do what everyone does: you open a query console and start writing SQL by hand, in the dark, under pressure, hoping you remember the schema.
The instinct after that night is to build another dashboard. Resist it. The problem is not that you're missing a dashboard. The problem is the whole idea of pre-building them.
A dashboard answers the question you had when you built it — never the one you have at 2 a.m.
That's the reframe. A dashboard is a frozen question. You picked the metrics, the grouping, the time window, the filters — all in advance, for a situation you were imagining, not the one you're in. Real incidents are specific in ways you can't anticipate. So teams try to anticipate harder: more dashboards, more filters, more drill-downs. The long tail of possible questions is effectively infinite, and you are trying to cover it by hand, one frozen question at a time. You will always lose that race.
What generative observability actually is
Stop covering the tail by hand. Build a layer that covers it on demand.
Generative observability is one addition to your stack: a natural-language command bar over your telemetry. You type the question you actually have — "why is extraction quality down this week?" — and instead of routing you to a pre-built view, it assembles a bespoke one, right then, for that question.
Under the hood it's four unglamorous steps:
- Interpret. An LLM turns your sentence into a structured intent: the metric (extraction accuracy), the window (last 7 days vs. prior), the cut you probably care about (by customer, by document type).
- Plan the views. From that intent it chooses a small layout — a couple of KPI tiles, a trend line, a breakdown by the dimension in question, an offenders table. Not "write one query" but "decide which handful of views, together, answer this."
- Query. It compiles that plan into read-only queries against your warehouse or telemetry store and runs them.
- Render. It drops the results into those view components and shows you a dashboard that did not exist ninety seconds ago and will not exist ninety seconds from now.
The LLM isn't inventing your data or "reasoning about your numbers." It's doing translation — English to a query plan to a layout — over telemetry you already collect. Whatever your team does to answer a question by hand at 2 a.m., you're teaching the system to do the boring first pass.
Why this beats pre-building
Three reasons, each one a thing a founder feels directly.
- It covers the long tail. You still can't predict the 2 a.m. question — but now you don't have to. The bespoke view is generated for the question instead of the question being crammed into whatever view happens to exist. The view fits the question, not the reverse.
- Anyone can ask. Today, "why is quality down for this customer?" is answerable only by whoever knows the schema and the query language — usually one or two engineers, usually the tired ones. A command bar turns that into a sentence a support lead or a PM can type.
- It's honest about what you don't know yet. A pre-built dashboard implies its metrics are the important ones. A blank command bar makes no such claim. It meets you where the incident actually is.
The proactive loop — where it becomes a weapon
On its own, a command bar is a nice convenience. Paired with your alerting, it becomes an on-call weapon.
Close the loop like this: an alert fires — "extraction quality dropped 13% in the last 24h." In most setups that's where the tooling stops and the human scramble begins. Here, the alert is a link. You click it, and the same generative layer receives the alert's context as a pre-filled question, and hands you the exact diagnostic view: the trend that shows when it broke, the breakdown that shows the drop is concentrated in one document type from one customer, the offenders table that surfaces the specific failing documents. Thirty seconds from page to probable cause.
Detect → ask → see → fix. That loop is the actual product. Detection alone just wakes you up. Detection wired to instant, question-shaped diagnosis is what turns a two-hour incident into a ten-minute one — which, for a small team on call, is the difference between a rough night and a ruined week.
The right-sizing guardrail
You still pre-build the handful of dashboards you look at every day. Generative observability is for the long tail, not the core. The four or five numbers you check with your coffee should be a fixed, fast, boring dashboard that loads the same way every morning. You do not want an LLM re-deriving your north-star metrics on every page load.
And you must bound it, hard. You are pointing a language model at a window into your data:
- Read-only, always. The query path can select. It cannot write, drop, or mutate.
- Scoped queries. Query a curated set of views or a read replica — not raw production tables — with row limits, cost ceilings, and timeouts.
- Metadata, not payloads. It reasons over the shape of your data — counts, rates, aggregates — not raw customer documents.
And don't build it before you have the telemetry to feed it. A generative layer over three log lines is worse than useless — it's confidently useless. First wrap your boundaries and get clean, structured telemetry flowing. Then put a command bar on top. Do it in that order or don't do it yet.
The payoff
The graveyard of dashboards was never a tooling problem. It was a category error — trying to pre-answer an infinite set of questions with a finite set of frozen ones. You don't win that by building faster. You win it by building the answering machine once, keeping your daily few, and letting the long tail generate itself.
What you get is a system you can talk to. An alert fires, you ask it what happened, and it shows you — in the shape of the question, not the shape of some dashboard you built for a different incident six months ago. Not fifty dashboards, but one layer that makes the fifty-first unnecessary.
Top comments (0)