There's a dashboard-shaped hole in most early-stage AI companies, and the usual instinct is to fill it with everything. Latency percentiles, token counts, cache hit rates, prompt versions, retrieval scores, a funnel, an NPS widget. Within a quarter you have forty charts and no idea which one to look at when something feels off.
The problem isn't too little measurement. It's that measurement without a decision attached to it is just decoration.
A metric earns its place on the dashboard only if a specific number would make you do a specific thing.
By that test, an early-stage AI startup needs about four. Here they are, in the order they'll save you.
1. Time-to-diagnosis
When an AI output is wrong, how long until you know why?
This is the metric that quietly governs your entire iteration speed, and almost nobody tracks it. A user says "the assistant gave me nonsense." Now the clock starts. Can you pull up that exact request, see the prompt that was assembled, the context that was retrieved, the model's raw output, and the tool calls it made — in minutes? Or does it take an afternoon of grepping logs and guessing?
For an AI product this is harder than for a normal CRUD app, because failures are probabilistic and contextual. The same prompt can succeed and fail. So you need the actual inputs preserved, tied together by a request ID that follows the user's action from the frontend all the way through your retrieval, inference, and post-processing steps.
Target: Under 10 minutes from "here's the complaint" to "here's the offending trace." Miss it and you're not shipping improvements — you're shipping guesses.
2. Cost per successful outcome
Are your unit economics real, or are you subsidizing every user?
Most teams track cost per token or cost per API call. Those are inputs, not economics. The number that matters is cost per outcome your user actually values — a resolved support ticket, an accepted draft, a completed workflow. That's the denominator that lines up with revenue.
Why "successful"? Because AI products spend real money on failures — the retry, the re-prompt, the multi-turn recovery, the human escalation. If you only count cost per API call you'll congratulate yourself on a cheap model while your effective cost per resolved ticket is 3× higher because a third of them need three attempts.
Target: Know this number, updated daily, with its trend week over week. Alert when it moves more than ~20% in a week.
3. Task success rate
Is the product actually working, from the user's point of view?
Uptime is not success. Your API can be at 99.9% while your assistant confidently hands users garbage. For an AI product, "working" is a quality judgment, so you need a lightweight, honest signal of it — thumbs up/down, an implicit signal like "did the user accept the suggestion or immediately redo it by hand," or a small offline eval set you run against every meaningful prompt or model change.
The trap here is over-engineering. You do not need a research-grade eval harness at ten customers. You need a signal directional enough to catch a regression before your users do.
Target: A success signal you trust enough to block a release on.
4. Lead time to production
How fast can a good idea become a live change?
Commit to production — capturing the whole pipeline: review, CI, deploy friction, and the invisible tax of "is it safe to ship right now?" For AI teams it matters double, because so much of your improvement is empirical. You don't know if the new prompt is better until it's in front of users. A slow, scary path to production means you run fewer experiments, which means you learn slower — the only thing you can't afford.
Target: Minutes-to-hours, not days. Multiple safe deploys a day.
The twenty you can ignore (for now)
Not because they're useless — because they don't yet change a decision at your stage. Detailed latency histograms, cache hit rates, per-endpoint error taxonomies, cohort retention curves, model-by-model A/B dashboards, infra utilization graphs. Every one of these becomes worth it eventually. Adding them now is how a four-person team ends up maintaining a monitoring stack instead of a product.
How they fit together
Notice these four form a loop, not a list:
- Task success rate tells you if there's a problem.
- Time-to-diagnosis tells you why, fast.
- Lead time tells you how quickly you can ship the fix.
- Cost per outcome keeps the whole loop honest about money.
That's the entire early-stage operating picture: is it working, why not, how fast can we fix it, and can we afford it. Four numbers. Each one attached to a decision. Everything else can wait until you've earned the right to complexity.
Top comments (0)