The most-quoted number in enterprise AI, and what it actually measures
You've seen it: "95% of AI pilots fail." It comes from MIT Project NANDA's July 2025 report, The GenAI Divide: State of AI in Business 2025, which found only 5% of organizations were translating AI pilots into real operational or financial impact.
Most people read that as "the models aren't good enough yet." That's not what the report says. In their words, the divide "does not seem to be driven by model quality or regulation, but seems to be determined by approach." And on the tools themselves: they "fail not because of poor models, but because they don't learn, adapt, or integrate. The lack of memory and feedback loops keeps GenAI stuck as a productivity enhancer, not a workflow transformer."
That reframes the whole thing as an architecture problem. Here's what that means concretely if you're the one building.
Three architectural failures hiding behind "the pilot didn't work"
1. No feedback capture at the decision point. Most deployments generate an output, a human accepts or rejects it, and that judgment evaporates. Nothing is stored about what the human changed or why. You've built a system that can't get better because you never wrote down whether it was right.
The fix is unglamorous: treat the human approval step as a structured data-capture event, not a UI gate. Log the original output, the final output, the delta, and (where you can get it) a reason code. That's a training signal and an audit record in the same row.
2. No durable memory across runs. A stateless call that re-derives context every time can't accumulate anything. Session memory isn't the same as institutional memory — what you want is a persistent, queryable record of past decisions that future runs can actually condition on.
3. Standalone instead of integrated. A chat window next to the workflow is not in the workflow. If a person has to leave the queue they live in, context-switch, prompt something, and paste a result back, you've added a step rather than removed one. The measurable-ROI deployments tend to be the ones where the agent operates inside the existing system of record.
The finding almost nobody quotes
Also in that report: back-office automation delivers stronger ROI than front-office functions, despite 50%+ of AI budgets going to sales and marketing. If you're picking where to spend build effort, that's a useful prior. Back-office work is typically higher-volume, more repeatable, and far easier to measure — which makes both the automation and the ROI case tractable.
In healthcare specifically, that's claims processing, prior authorization, denials and appeals, fraud review. Not the flashy demos — the queues.
Why human-in-the-loop designs sidestep this by accident
Worth naming: MIT studied enterprise GenAI broadly and doesn't endorse any particular approach or vendor. But if you squint at their failure pattern — no memory, no feedback, no integration — a governed agent design avoids all three almost incidentally. The approval step is your feedback capture. The audit trail is your durable memory. And an agent that has to route work to a named reviewer necessarily lives inside the workflow rather than beside it.
We build this pattern at IntelliBooks Studio for healthcare back-office workflows. Happy to get into the feedback-schema design or how we structure the decision log in the comments.
Top comments (0)