The 12 Questions I Run Before I Let Any Team Build an AI Feature
Most failed AI projects don't fail at implementation. They fail before a single line of code gets written, when somebody answers "should we?" with "everyone else is."
I run an audit before any AI build kicks off — for my own work, and (when people pay me to) for theirs. It's twelve questions. None of them are technical. By question 8 most people have already changed what they want to build, and by question 12 about a third have decided not to build anything at all. That's the point.
Here's the full list, what each question is actually testing for, and the failure mode it catches.
1. What's the manual version of this workflow today?
If nobody can describe the manual version end-to-end, the AI version will be a hallucinated process pretending to be a product. You can't automate something you can't draw on a whiteboard.
Failure caught: Building automation for an imagined workflow nobody actually performs.
2. How often does this workflow run, and who triggers it?
Daily? Weekly? On-demand by one person, or twenty? "How often" tells you whether to build a scheduled job, an API, a chat interface, or nothing at all. "Who triggers it" tells you the user surface.
Failure caught: Building a Slack bot for a workflow that runs twice a quarter.
3. What's the cost of getting it wrong silently?
Every LLM-based system fails silently sometimes. Wrong summary, wrong tag, wrong recipient. Ask: if this gives a confidently wrong answer and nobody notices for two weeks, what's the damage?
If the answer is "we lose a customer / we ship the wrong product / we send the wrong invoice" — you don't have an AI problem, you have a review-loop problem. Solve that first.
Failure caught: Deploying autonomous agents into workflows where silent failures compound.
4. What does the human currently do that the model can't?
Be specific. "Judges quality" — based on what signals? "Knows the customer" — from which data? If you can't enumerate what the human knows that the model doesn't, you can't tell whether the model can replace them, augment them, or shouldn't be involved.
Failure caught: Replacing tacit expertise with a model that has no access to the inputs that expertise depends on.
5. What's already structured, and what's still tribal knowledge?
LLMs work best when context is already structured somewhere — docs, tickets, a CRM, a database. If the relevant knowledge lives in three people's heads and a Slack DM from 2024, the first project isn't AI. It's documentation.
Failure caught: Trying to RAG over an empty knowledge base.
6. What's the input format, and is it consistent?
Show me ten real examples of the input. Are they the same shape? Same length? Same vocabulary? If your inputs are wildly variable PDFs, screenshots, voice notes, and free-text emails, your first build problem is ingestion, not intelligence.
Failure caught: Treating "make AI parse this" as one project when it's actually three.
7. What does success look like in numbers?
"Saves time" is not a metric. "Reduces ticket triage from 12 minutes to under 90 seconds, with <5% misroute rate, measured over 200 tickets" is a metric. Without numbers, you'll never know if it worked, and the system will quietly drift.
Failure caught: Shipping AI features that nobody can prove are doing anything.
8. Who maintains it on a Tuesday in 18 months?
The model gets deprecated. The prompt drifts. The downstream API changes. Who's the named human who fixes it when it breaks at 4 PM on a Tuesday in mid-2027?
If the answer is "the consultant who built it" or worse, "we'll figure it out" — you're not buying a system, you're renting a problem.
Failure caught: Orphaned AI features that quietly degrade until someone notices customers complaining.
9. What happens when the model says "I don't know"?
Most teams design the happy path and forget the abstain path. What does the system do when confidence is low? Escalate to a human? Refuse? Default to a safe fallback? "Try anyway and hope" is not an answer.
Failure caught: Confidently wrong outputs in cases the system was never designed to handle.
10. What data leaves your environment, and where does it go?
Customer names? Internal financials? Source code? Each LLM call is a data egress. If you can't draw the line on a whiteboard from "user input" to "third-party API" to "logged where," you're not ready for compliance review, never mind procurement.
Failure caught: Surprise data-exfiltration findings six months in, when legal finally looks at it.
11. What's your kill switch?
If the AI feature starts misbehaving in production at 2 AM, what turns it off? A feature flag? A config push? A code deploy? "We'd have to roll back" is the wrong answer. Build the off switch before the on switch.
Failure caught: Public AI failures that stay public for hours because nobody planned for the rollback path.
12. If we don't build this, what breaks?
This is the most important question, and the one that kills the most projects. If the honest answer is "nothing, we just thought it'd be cool" — congratulations, you've saved yourself six months and a six-figure budget.
The best AI projects answer this with a specific, painful, expensive problem. The worst ones answer with "innovation" or "staying competitive."
Failure caught: Every AI project that exists because someone read an article on a plane.
How to use this list
Run it before scoping. Run it again before kickoff. If you can't answer a question, that's not a sign to skip it — it's a sign you've found the actual first deliverable.
Two patterns I see consistently:
- Teams that can't answer 1, 4, or 5 need a discovery / documentation phase, not an AI build.
- Teams that can't answer 8, 10, or 11 need an ops review with security and SRE before anyone touches a model.
Most of the wins in AI in 2026 aren't model-level wins. They're scoping wins. The teams that ship working AI systems aren't smarter — they just answered question 12 honestly before they started.
If you'd rather not run this audit yourself, this is roughly the framework I use for paid strategy sessions — 60-minute workflow audit, top 5 ROI-ranked automation opportunities, written strategy doc in 24 hours. The most common outcome is that we cut the original scope by half and ship something that actually works, instead of something that demos well.
But you don't need me. You need question 12.
Top comments (0)