Most enterprise AI projects don't fail because the model was bad. They stall because nobody agreed on what "done" means, the data lives in a system no one can reach, or the security review shows up in week ten.
Questions are cheap in week zero and expensive in week twelve. Here are seven I'd want answered before anyone opens an IDE, with what a good answer looks like and what should make you pause.
1. What problem are we solving, and how will we know it worked?
"We need an AI strategy" is not a problem. "Tier-1 support agents spend too long routing tickets" is.
A good answer names one workflow, one team that feels the pain, and a number you can measure today. If you can't state the baseline (time per task, error rate, cost per case), you can't prove the project helped, and the first budget review will be painful.
Good answer: a single use case, a baseline metric, and a target someone has agreed to be held to.
Red flag: the success criteria are "improve efficiency" or "get ahead of competitors."
2. Is our data actually ready?
Teams routinely discover this one halfway through the build. Before committing, check:
- Access: can the system read the data through an API, or is it locked in exports and spreadsheets?
- Quality: is it complete and consistent enough to trust, or does it need weeks of cleanup?
- Freshness: how stale is it by the time the AI sees it?
- Permissions: does the AI inherit the same access rules a human user would have?
- Ownership: is there a named person who can fix the source when it's wrong?
An AI agent that confidently answers from outdated or over-shared data is worse than no agent at all.
Good answer: the sources are named, reachable, and owned.
Red flag: "the data is in a few places, we'll figure it out."
3. How will this connect to the systems we already run?
Enterprise AI rarely lives alone. It has to read from the CRM, write to the ticketing system, and authenticate through your existing identity provider.
Ask early:
- Which systems does it need to read from, and which will it write to?
- Do those systems have stable, documented APIs, or are we wrapping something legacy?
- How will the AI authenticate, and with what level of privilege?
- Is there a gateway or integration layer, or will every agent build its own point-to-point connections?
Point-to-point integrations feel fast in a pilot and become unmaintainable at the third agent. A shared integration layer costs more up front and pays for itself quickly.
Good answer: a list of systems, how each is accessed, and a least-privilege auth plan.
Red flag: the integration plan is "use the admin credentials for now."
4. Who owns it, and who is accountable when it's wrong?
AI systems will make mistakes. The question is whether anyone is responsible for catching them.
Decide in advance:
- Who is the product owner, and who is on call when it misbehaves?
- Which decisions can the system make alone, and which need a human to approve?
- What does escalation look like when the confidence is low?
- How do users report a bad answer, and where does that feedback go?
Human-in-the-loop isn't a sign of a weak system. It's usually what lets you ship the first version at all.
Good answer: a named owner, a clear autonomy boundary, and an escalation path.
Red flag: "the model will handle it."
5. What are our security, compliance, and governance requirements?
Bring security and compliance in during discovery, not at the launch gate. The questions are different from traditional software:
- What data will reach the model, and does any of it include PII or regulated information?
- Where is that data processed and stored, and does residency matter for us?
- Which models and providers are approved?
- What gets logged, and can we audit why the system made a given decision?
- How do we handle prompt injection and untrusted input when the agent can take actions?
If your organization already has an AI policy, read it first. If it doesn't, expect these questions to become the policy.
Good answer: requirements are written down, and security has seen the architecture.
Red flag: compliance hasn't heard about the project yet.
6. What will it cost to run, and to keep running?
The prototype's bill is not the production bill. Budget for the whole lifecycle:
- Inference cost per request, at the volume you expect after launch
- Evaluation, monitoring, and logging infrastructure
- Ongoing prompt, model, and data maintenance as things drift
- The engineering time to support it, which is usually the biggest line
Work out the unit economics. If one automated task costs more than the manual one it replaces, you need to know that before scaling, not after.
Good answer: a cost per request, a monthly ceiling, and a comparison with the current process.
Red flag: the only number anyone has is the pilot's API bill.
7. How do we get from pilot to production, and what's our exit plan?
Plenty of pilots impress in a demo and die in review because no one planned the path forward. Agree on these up front:
- Evaluation: a labeled test set and a pass bar, so "it seems good" becomes a measurement
- Observability: traces, quality metrics, and alerts once real users are involved
- Rollback: a feature flag or kill switch that works in minutes, not days
- Portability: how hard it would be to swap the model or vendor later
- Stop criteria: the result that tells you to cancel, which is as important as the one that tells you to scale
Good answer: defined evaluation, a rollback path, and written exit criteria.
Red flag: "we'll productionize it after the demo."
Put it on one page
Before kickoff, turn the answers into a short brief that everyone signs off on. Something as light as this works:
project: support-ticket-triage
problem: Tier-1 agents spend too long routing tickets
success_metric:
baseline: <median time to first routing today>
target: <number the team commits to>
measured_by: <dashboard or report>
data:
sources: [ticketing system, knowledge base]
owner: <named person>
contains_pii: true
integration:
systems: [ticketing API, SSO]
auth: service account, least privilege
ownership:
product_owner: <name>
on_call: <team>
human_review: low-confidence routes
governance:
data_residency: <region>
audit_log: required
approved_models: [<list>]
cost:
per_request_budget: <amount>
monthly_ceiling: <amount>
pilot_to_prod:
eval_set: <size and source>
rollback: feature flag
stop_criteria: <what makes us cancel>
The blanks are the point. Every <placeholder> you can't fill in is a risk you've found for free.
How to use this list
Run it as a one-hour workshop with engineering, security, and the business owner in the same room. If you can't answer most of the seven, that isn't a reason to abandon the project. It's a sign the next step is discovery, not development.
The teams that scale AI successfully tend to be the ones that did the boring questions first.
Which of these has caught your team off guard? Share it in the comments.
Top comments (0)