This article was originally published on davidohnstad.net. I cross-post here to reach the Dev.to community.
Why Most Enterprise AI Pilots Never Escape the Proof-of-Concept Phase
We built an AI-powered anomaly detection feature for a customer health monitoring dashboard. Three engineering sprints. One PhI-level data scientist. Two weeks after release, usage metrics showed the feature was toggled off by 89% of account owners within the first login session. The feedback wasn't about accuracy—the model worked. The problem was that we'd inserted a sophisticated prediction engine into a workflow where users needed fast triage decisions, not probabilistic explanations. According to Gartner's 2024 State of AI report, this pattern holds across the industry: 54% of AI features in enterprise software get disabled or ignored within 90 days of deployment, not because the AI fails technically, but because the feature was never mapped to an actual user decision point.
This article introduces a practitioner framework for evaluating which AI capabilities should actually make it into your enterprise software roadmap. The challenge right now isn't whether to invest in AI—that decision has already been made for most organizations during Q1 budget cycles. The challenge is figuring out which AI capabilities to prioritize in Q3 when your backlog has fifteen competing "AI-enhanced" feature requests and your product team is being asked to justify the ROI of each one. This is a named, step-by-step decision model David Ohnstad uses to filter AI feature ideas down to the ones that will actually get adopted, not just launched.
The Stakes: When AI Investment Becomes Technical Debt Instead of Product Value
The cost of shipping the wrong AI feature isn't just the engineering time spent building it. It's the organizational credibility lost when users learn to ignore anything labeled "AI-powered" in your product. One SaaS company David worked with launched three separate AI features in an 18-month window. Each one was technically sound. Each one solved a problem nobody had prioritized. By the third launch, internal adoption surveys showed customer success teams were actively advising new users to skip the AI sections of the platform "until they're more useful." That's not a product velocity problem—that's a product trust problem, and it compounds.
Research from McKinsey's 2025 AI in Enterprise Software study found that organizations with three or more underused AI features saw a 31% drop in adoption rates for subsequent AI launches compared to companies that shipped fewer, more targeted AI capabilities. The takeaway: users form expectations about whether your AI features are worth their time based on the cumulative track record, not individual launches. Ship two AI features that miss the mark, and your third feature—even if it's genuinely valuable—starts with a credibility deficit.
The typical failure pattern looks like this: a product team identifies an area where machine learning could theoretically add value, secures engineering resources, builds the feature, launches it with internal fanfare, and then watches usage flatten within eight weeks. Post-mortems usually cite "change management" or "user education gaps," but those explanations miss the real issue. The feature wasn't designed around a decision the user was already trying to make. It introduced a new capability without removing friction from an existing workflow. David Ohnstad's data product management writing consistently emphasizes this principle: AI features that require users to change their behavior to accommodate the feature are dead on arrival. The feature must accommodate the user's existing decision process.
The Decision-First AI Capability Stack
This is a four-layer evaluation model David Ohnstad uses when a stakeholder proposes adding AI to an enterprise software feature. The name captures the core principle: you start with the decision, not the technology. Every AI capability must map to a specific decision a user is already making—and it must make that decision faster, more confident, or less error-prone. If you can't name the decision in one sentence, the AI feature isn't ready for the roadmap yet.
Layer 1: Decision Mapping—Identify the exact decision point where the AI capability would intervene. Not the general workflow. The specific moment where a user has to choose between Option A and Option B, or decide whether to act or wait. For the anomaly detection feature mentioned earlier, the decision point wasn't "understand customer health trends"—it was "should I escalate this account to a senior CSM today or wait another week?" Once you name the decision that specifically, you can test whether the AI output maps to it. In that case, it didn't. The model provided confidence scores and contributing factors, but users needed a binary recommendation with reasoning they could forward to their manager. The mismatch between model output and decision structure killed adoption.
Layer 2: Data Readiness Audit—Verify that the data required to train and run the AI model is already being collected, cleaned, and structured in your system. This is the step most teams skip because it feels like an engineering concern, not a product prioritization concern. But data readiness determines whether you're six weeks from launch or six months. David encountered a case where a recommendation engine feature was approved in Q1, only to discover in Q2 that the event tracking needed to train the model wasn't instrumented in the product yet. By the time the data pipeline was built and had collected enough historical data to train a minimally viable model, the launch window had shifted by two quarters. The feature missed the fiscal year it was budgeted for. The lesson: if the data isn't already flowing, the AI feature isn't ready for your next planning cycle.
Layer 3: Output Integration Test—Design a mockup of how the AI output appears in the user interface and test whether it integrates into the existing workflow without requiring the user to context-switch. This is where the "AI as a sidebar" mistake happens. Teams build sophisticated models, then display the output in a separate panel or modal that users have to deliberately navigate to. If the AI insight requires an extra click or a tab switch, adoption drops by an order of magnitude. The integration test asks: can the user act on this AI output without leaving the screen they're already on? For David Ohnstad Minnesota-based enterprise teams, this often means embedding AI recommendations directly into tables, dashboards, or notification streams—not creating a new "AI Insights" section users have to remember to check.
Layer 4: Reversibility and Transparency Design—Build in the ability for users to see why the AI made a recommendation and override it without friction. This is the counterintuitive step. Most AI product teams focus on accuracy and confidence scores, assuming that a 95% accurate model will earn user trust. But enterprise users don't adopt AI features because the model is accurate—they adopt them because they can verify the reasoning and reverse the decision if needed. One procurement software company David worked with added a "Show me why" button to every AI-generated vendor risk score. Usage of the AI feature jumped 40% in the first month after that button was added, even though the underlying model didn't change. The transparency feature gave users confidence that they weren't blindly trusting a black box, which made them more willing to rely on the AI output for lower-stakes decisions.
Why This Approach Fails in Organizations That Treat AI as a Platform Capability
The Decision-First AI Capability Stack breaks down in companies that centralize AI development in a dedicated machine learning team separate from product development. The structural problem is that ML teams are typically measured on model performance—accuracy, precision, recall—not on whether the feature gets adopted. This creates a misalignment where the ML team optimizes for the wrong success metric. David saw this play out at a company where the ML team built a churn prediction model with 92% accuracy, then handed it off to the product team to "find a use case for it." The product team embedded it in a customer health dashboard, but the prediction timeline didn't match the intervention window customer success teams actually used. The model predicted 90-day churn risk, but CSMs intervened at the 30-day mark. By the time the model flagged an at-risk account, the intervention opportunity had already passed. The model was accurate. The product integration was useless.
The fix isn't better collaboration between ML and product teams—it's changing the organizational structure so that AI capabilities are developed inside product squads, not handed down from a centralized ML team. That requires product managers who can write SQL, understand data pipelines, and collaborate directly with data scientists. It also requires data scientists who are willing to iterate on model design based on user feedback, not just optimize for algorithmic performance. This is where AI & Machine Learning in Enterprise Software becomes a product management skill, not just a technical capability. The best AI features David has shipped were built by cross-functional squads where the PM, designer, engineer, and data scientist sat together and co-designed the feature from the decision point backward to the model architecture.
Stop Adding AI Features to Boost Product Differentiation—Most Buyers Don't Care Yet
Here's the contrarian claim: enterprise software buyers in mid-market and SMB segments do not yet assign higher willingness-to-pay to products with AI features, and adding "AI-powered" to your feature list will not increase win rates in competitive evaluations. According to Forrester's 2024 B2B Software Buyer Behavior Report, only 18% of mid-market buyers cited AI capabilities as a top-three evaluation criterion when comparing SaaS platforms, and that figure dropped to 11% for SMB buyers. The conventional wisdom in product strategy right now is that you need AI features to stay competitive. The data says otherwise—at least for now. What buyers care about is whether your product solves their workflow problem faster than the alternative. If the AI feature doesn't visibly reduce time-to-outcome, it's not moving the competitive needle.
This doesn't mean you should ignore AI. It means you should stop prioritizing AI features as differentiation plays and start treating them as workflow optimization investments. The ROI case for an AI feature should be measured in minutes saved per user per week, not in "competitive positioning" or "innovation narrative." One financial services software company David advised wanted to add AI-driven document classification to their platform because two competitors had launched similar features. When they ran a pilot with 50 users, the time savings averaged 90 seconds per document—but users processed an average of 3 documents per day. The total time saved per user per week was 7.5 minutes. The feature cost eight engineering months to build. That math doesn't close. The company shelved the feature and redirected the engineering capacity toward a bulk document upload feature users had been requesting for two years. That feature increased user satisfaction scores by 22 points in the next NPS survey. Boring infrastructure work beat the shiny AI feature because it solved a bigger workflow friction point.
What Senior Product Leaders Should Audit in Q3 Planning Cycles
If you're heading into second-half planning and your roadmap includes multiple AI feature requests, run this audit before committing resources. First, for each proposed AI feature, write down the specific user decision it supports in one sentence. If you can't, the feature isn't scoped yet—send it back for more discovery work. Second, calculate the current time-to-decision for that workflow without AI, then estimate the time-to-decision with the AI feature. If the gap is less than 30% time savings, the feature probably won't drive measurable adoption. Third, identify whether the data required to train and run the model is already instrumented in your product. If it's not, add six months to your timeline estimate and budget for data pipeline work before the AI engineering starts.
The hardest part of this audit is saying no to AI features that are technically impressive but strategically misaligned. David has watched senior product leaders approve AI projects because they sounded innovative or because a competitor launched something similar, even when the internal data showed the feature wouldn't move core product metrics. The discipline required here is treating AI features like any other product investment: they must have a clear hypothesis, a measurable outcome, and a feedback loop that tells you within 90 days whether the feature is working. If you wouldn't ship a non-AI feature without that rigor, don't lower the bar just because machine learning is involved.
How do you prioritize AI features when you have limited data science resources?
Start with the Decision-First AI Capability Stack and filter ruthlessly at Layer 2: Data Readiness. If the data required to train the model isn't already instrumented and flowing in your product, the feature moves to a future roadmap cycle. Focus your limited data science capacity on features where the data exists today and the user decision is already happening in your product workflow. This approach ensures you're building AI features that can launch within a single quarter instead of multi-quarter science projects that may never ship.
What's the difference between an AI feature that gets adopted and one that gets ignored?
Adopted AI features integrate directly into the workflow the user is already completing—they don't require context-switching or navigating to a separate section of the product. Ignored AI features live in sidebars, dedicated dashboards, or "Insights" tabs that users have to remember to check. The integration difference determines whether the AI becomes part of the user's daily routine or becomes optional exploration they skip under time pressure. Adoption follows the path of least resistance.
Why do enterprise buyers say they don't prioritize AI features in software evaluations?
Most enterprise buyers have experienced AI features that overpromised and underdelivered—either the accuracy wasn't reliable enough to trust, or the feature didn't map to their actual workflow. Forrester's 2024 research shows buyers now treat "AI-powered" as marketing language rather than a functional differentiator. Buyers care whether your product solves their problem faster than alternatives. If the AI visibly reduces time-to-outcome, it becomes a competitive advantage. If it doesn't, the "AI" label is irrelevant to the purchase decision.
Two Closing Takeaways and One Uncomfortable Question
For practitioners: the best AI feature you can ship this quarter is probably the one that automates a repetitive decision your users are already making manually 20 times a day. It's not the most technically sophisticated model—it's the one that saves the most cumulative time across your user base. For leaders: stop approving AI features based on competitive pressure or innovation narratives. Require the same ROI rigor for AI investments that you apply to non-AI product work. If the feature can't demonstrate measurable time savings or error reduction within 90 days, it doesn't belong in this planning cycle.
Here's the question: when you review your current roadmap, how many of your planned AI features are there because they solve a user decision problem you've directly observed, versus how many are there because someone said "we should be doing more with AI"? If the ratio leans toward the second category, you're building for the wrong reasons—and your adoption metrics six months from now will prove it.
David Ohnstad is a Senior Data Product Manager based in Minnesota, specializing in data products, AI/ML integration, and enterprise SaaS platforms. Follow his work at github.com/davidohnstad40-netizen.
Top comments (0)