Organizational structure determines AI outcomes more than technology ever will
McKinsey’s research found that more than 80% organizations are not yet seeing a tangible impact on enterprise-level EBIT from AI and Agents. This suggests that while adoption is broadening, most companies are still struggling to turn AI and Agents into scaled financial results.
But there is an important piece of the story that is missing. A separate analysis of 140 enterprise AI implementations found that 77% of failures were organizational in nature, with technical issues like model performance, data quality, and integration complexity accounting for less than a quarter.
Your org chart is the first system AI has to survive before it reaches a single customer or workflow, and these five structural mistakes consistently prevent it from getting there.
1. Your Chief AI Officer Reports Nowhere Near the P&L
The 2026 AI & Data Leadership Executive Benchmark Survey found that 38.5% of companies have now appointed a Chief AI Officer or equivalent, but there’s almost no consensus on where that role sits. Reporting lines are split across technology, business, and transformation leadership, with no dominant model emerging and no clear pattern connecting any one reporting structure to better outcomes.
That fragmentation carries real downstream consequences. When AI leadership reports into a CTO or CIO function, the role tends to optimize for infrastructure and tooling decisions rather than business impact. When it reports into a transformation office, it gravitates toward strategy decks and governance frameworks that rarely survive contact with operational reality.
Neither path connects AI or Agents directly to revenue, margin, or operational throughput, which means the person nominally responsible for AI results often has no line of sight into the metrics that define them.
2. Your AI or Agent Team Lives in IT Instead of in the Business
When AI or Agent capability gets housed inside the IT department, it inherits IT’s entire operating model, meaning projects get scoped through a service request lens, prioritization follows the IT backlog, and success gets measured in uptime and deployment velocity rather than business outcomes.
This is a fundamental structural mismatch. AI is a business capability that requires technical infrastructure, and the distinction matters because AI initiatives that start with a business problem and work backward toward the right technical approach tend to survive past the pilot stage, while initiatives that start with a model and go looking for a use case tend to stall indefinitely.
Organizations running AI teams embedded within business units, or at minimum co-located with business leadership, consistently outperform centralized IT-led models on both adoption and value delivery.
3. Your Steering Committee Owns Accountability for Nothing
AI steering committees are one of the most popular governance structures in enterprise AI programs, and they’re also one of the least effective.
The typical setup includes senior representatives from multiple functions who meet monthly to review progress, offer guidance, and align priorities, but in practice, these committees almost always devolve into a venue for status updates where no actual decisions get made.
The root issue is accountability without power. Steering committees rarely control budget allocation, staffing decisions, or deployment timelines, which means they can recommend changes but have no mechanism to compel them. When an AI initiative hits an organizational obstacle (and every one does), the committee discusses it, documents it, and then waits for someone else to resolve it, creating a governance layer that absorbs time without reducing friction.
Research on AI governance maturity from McKinsey’s 2026 AI Trust Maturity Survey reinforces how widespread this gap is, with only about 30% of organizations reaching a maturity level of three or higher in governance, even as their technical and data capabilities continue to advance. The organizational decision-making apparatus simply hasn’t kept pace with the technology it’s supposed to govern.
4. You Built AI Skills in One Team and Called It Done
Concentrating AI talent in a single team feels efficient at first, but the problems with this approach emerge at scale. When every AI initiative has to flow through the same team, that team becomes a bottleneck.
This pattern appears so frequently in enterprise organizations that it has earned a name in organizational design circles. It’s called the Center of Excellence trap.
The CoE starts as a strategic asset and gradually evolves into a capacity constraint that chokes the very pipeline it was built to open. A CIO article from late 2025 described the resulting dynamic well, noting that business units inevitably branch off on their own when the central AI team can’t keep pace, creating fragmented and ungoverned efforts scattered across the company with no shared standards or oversight.
The more sustainable model is capability distribution. Instead of hoarding AI expertise in one group, the investment goes into building baseline AI literacy and applied skills across functions. This allows the central team to shift from doing the work to enabling others to do it by providing tooling, standards, training, and quality guardrails while the business units own execution and outcomes.
5. Your Center of Excellence Has No Authority to Make Anything Stick
This is the inverse of mistake four. Some organizations do build a Center of Excellence with a genuine mandate to drive AI adoption across the enterprise, staffing it well, giving it a clear charter, and expecting it to set standards for how AI gets developed, deployed, and monitored. Then they forget to give it any enforcement power.
What follows is predictable. The CoE publishes best practices that business units ignore, develops governance frameworks that project teams route around, and recommends tooling standards that departments override. Without budget influence, or the organizational standing to block non-compliant deployments, the CoE becomes an advisory function that advises no one in particular and enforces nothing at all.
This is a design failure at the leadership level. A CoE with clear standards but no enforcement mechanism creates the illusion of governance while fragmented, uncoordinated AI adoption continues underneath it.
The Real Infrastructure Problem
These five mistakes share a common thread. They all treat AI as something that can be added to an existing organizational structure without redesigning how decisions get made, who owns outcomes, and where authority actually lives.
AI underperformance in most organizations traces back to an org chart that was built for a different kind of work and never updated to reflect how AI-driven operations actually need to function.
The companies capturing real returns in 2026 are the ones willing to redesign reporting lines, redistribute decision rights, and place AI leadership where it can actually influence how the business operates on a daily basis.
If you’re reviewing your AI strategy this quarter, start with the org chart. The structure you’re running determines the ceiling of what AI can deliver, and right now, most ceilings are set lower than anyone realizes.
…
Nick Talwar is a CTO, ex-Microsoft, and a hands-on AI engineer who supports executives in navigating AI adoption. He shares insights on AI-first strategies to drive bottom-line impact.
→ Follow him on LinkedIn to catch his latest thoughts.
→ Subscribe to his free Substack for in-depth articles delivered straight to your inbox.
→ Watch the live session to see how leaders in highly regulated industries leverage AI to cut manual work and drive ROI.
Top comments (0)