“Could an LLM do this?” is usually the easiest question in an AI product discussion. A more useful question is: “Does probabilistic generation improve this user outcome enough to justify its new failure modes and operating cost?”
The distinction prevents an impressive prototype from quietly becoming a fragile product dependency.
Define the job without mentioning AI
Start with a sentence that describes the user, trigger, and result:
When a support lead receives a long escalation, they need a draft timeline with links to the source messages so they can prepare a review faster.
This is testable. “Add an AI copilot to support” is not.
Then name the current alternative: manual work, search, a template, deterministic code, an existing vendor feature, or doing nothing. An LLM is competing with that baseline, not with an empty screen.
Check whether generation is actually needed
LLMs are a plausible fit when inputs are unstructured, acceptable outputs have more than one valid form, and semantic transformation creates value. Examples include drafting, summarization, classification with ambiguous language, and question answering over permitted text.
Prefer deterministic software when the task requires exact arithmetic, stable policy enforcement, authorization, database constraints, or a fixed mapping that ordinary code can express. A model can help interpret a request, but it should not become the source of truth for a refund limit or access rule.
A useful decomposition is:
interpretation → retrieval → decision → action → explanation
Only some stages may need a model. Keeping the other stages deterministic reduces cost and makes failures easier to contain.
Score the opportunity on five dimensions
Use a short written assessment rather than one blended “AI potential” score.
| Dimension | Question | Evidence to collect |
|---|---|---|
| User value | Does this remove meaningful time or friction? | task frequency, baseline completion time, user interviews |
| Output tolerance | Can a user detect and correct a weak result? | severity classes, review workflow, reversibility |
| Data readiness | Is the required context available and permitted? | source ownership, freshness, access controls, retention rules |
| Evaluation | Can the team recognize an acceptable result? | labeled cases, rubrics, prohibited outcomes |
| Operations | Can the team support latency, cost, and incidents? | volume model, timeout budget, fallback, owner and runbook |
A feature with high user value but no reliable evaluation is a research project. A feature with good model performance but low task frequency may still have poor economics.
Compare three implementation levels
Do not jump from “manual” to “fully autonomous.” Compare at least these levels:
- Assist: generate a draft that a user explicitly reviews.
- Recommend: propose a structured decision or action with evidence.
- Act: execute within bounded permissions, with confirmation or rollback where needed.
The assist version often creates most of the learning with less downside. Moving toward action should require stronger authorization, monitoring, and evidence—not merely a better prompt.
Build the total-cost model
Per-token price is only one term. Estimate:
monthly cost =
model inference
+ retrieval and storage
+ evaluation runs
+ observability
+ human review
+ support and incident response
+ expected rework from incorrect output
Model at least a typical and high-volume scenario. Include retries and long inputs. Use the provider’s current pricing in a maintained configuration because published prices and model names change.
Also estimate the cost of the baseline. If a draft saves two minutes but adds a mandatory three-minute review, the prototype’s apparent speed is not product value.
Set launch gates before the prototype wins hearts
Write acceptance criteria while the team can still stop:
- a target user segment completes the defined task faster or more successfully than the baseline;
- critical error classes are absent from a representative evaluation set;
- users can inspect sources or relevant input when factual grounding matters;
- the feature has a timeout, cancellation path, and understandable failure state;
- private data access follows existing application permissions;
- cost and latency stay within agreed budgets at expected volume;
- an owner can disable or fall back from the feature without an emergency release.
Do not reduce all quality to one average score. A summarizer that is usually fluent but occasionally invents a security action needs a separate metric for that severe failure.
Measure adoption as a funnel
“AI button clicked” is not adoption. Track a sequence:
eligible task
→ feature started
→ usable result returned
→ result accepted or meaningfully edited
→ task completed
→ user returns for a similar task
Pair product events with qualitative feedback. Heavy editing can mean healthy collaboration, poor output, or a user who needs more control. The event alone cannot tell you which.
Measure overrides and abandonment without punishing them. A visible correction path is a safety feature, and users should not be nudged to accept a weak suggestion just to improve an internal metric.
Revisit build, buy, and remove
The initial decision is not permanent. Re-evaluate when volume, model quality, policy, pricing, or available product features change.
- Buy when a vendor solves a non-differentiating job with acceptable controls and integration cost.
- Build when the workflow, data boundary, or evaluation capability is strategically distinctive.
- Remove when measured value does not clear the ongoing operational burden.
NIST’s AI Risk Management Framework organizes AI work around governing, mapping, measuring, and managing risk. Its Generative AI Profile adds risks and actions specific to generative systems. They are useful inputs for product gates, especially when a feature touches consequential decisions or sensitive data.
The best LLM feature is not the one with the most autonomy. It is the smallest probabilistic component that produces a measurable user benefit inside a system the team can evaluate, operate, and stop.
Top comments (0)