By Paul Okhrem · paul-okhrem.com
If you're a CEO trying to figure out which AI consulting partner to work with, you're navigating a market that has approximately no useful signal-to-noise ratio.
Every firm has "deep AI expertise." Everyone has case studies. The PowerPoint decks are indistinguishable. Prices vary by a factor of ten with no obvious correlation to quality. And the space has attracted enough opportunists — consultants who rebranded last year after years in a completely different domain — that due diligence is genuinely difficult.
I put together this resource because I kept having the same conversation with CEOs who were either about to make an expensive mistake or already trying to recover from one.
The three categories you're actually choosing between
The market segments into three broad types, and they serve different needs.
Strategy-only consultants help you think about AI — where it applies in your business, what the roadmap should look like, how to build internal capabilities. They typically don't build anything. If you need to think before you build, they're valuable. If you already know where you're going and need implementation, they'll cost you time.
Implementation shops build the thing. These range from boutique technical firms to staffing-adjacent shops that place contractors. Quality varies enormously. The key question is whether they have domain experience in your industry or if they're AI generalists — and whether the people who sell the engagement are the people who will do the work.
Embedded advisors / fractional operators sit somewhere between strategy and implementation — they advise, but they also roll up their sleeves, join your leadership conversations, and help build internal capability alongside any external vendors. This model tends to work best for mid-market companies that aren't ready to hire a full-time CAIO but need someone with real skin in the game.
Questions that actually differentiate
Most CEOs ask the wrong questions in vendor evaluations. "Tell me about your AI capabilities" produces a slide. These questions produce information.
"Walk me through a project where the AI implementation didn't achieve its initial goals. What happened and what did you do?"
Every honest practitioner has these stories. A firm that only tells success stories hasn't done enough work, or isn't being straight with you. What you're listening for: intellectual honesty, a clear diagnosis of what went wrong, and evidence that they adapted.
"Who specifically will be working on this engagement, and can I meet them before we sign?"
The bait-and-switch is real in consulting. Senior partners sell, junior staff deliver. Ask to meet the actual team. If the answer is vague, treat it as a red flag.
"What does success look like at 90 days, and how will we measure it?"
If they can't answer this precisely for your specific context, they don't know your business well enough yet — which is fine early in a conversation, but should be resolved before you sign. Vague success criteria protect the consultant, not you.
"What internal capabilities should we have built by the end of this engagement that we don't have now?"
The goal of external consulting should be to make itself less necessary over time, not more. If the answer focuses entirely on deliverables and not on knowledge transfer, you're building dependency, not capability.
"What AI projects do you think are overhyped for companies at our stage?"
The answer reveals intellectual honesty and whether they're willing to tell you things you might not want to hear. Consultants who agree with everything you say are not useful.
Red flags in the selection process
They lead with the technology, not the problem. "We'll use LLMs and agentic AI and RAG" is not a strategy. The technology should follow from the problem, not the other way around.
The case studies are from different industries with no explanation of relevance. Transferable lessons are real, but they require explicit bridging. "We did this for a bank, and here's why it applies to your manufacturing context" is credible. A logo slide is not.
The pricing is purely time-and-materials with no outcome accountability. Pure T&M consulting has no skin in the game. Outcome-based components — even partial ones — align incentives better.
They can't explain the limitations of their recommended approach. Every AI approach has failure modes and constraints. A consultant who can't articulate these clearly either doesn't understand the technology or is overselling you.
The proposal arrived very fast. A thoughtful proposal takes time. A generic proposal delivered in 24 hours tells you they weren't listening in the conversation — they were retrieving a template.
What to actually check in references
References provided by the vendor will be curated. Make them useful:
Ask the reference: "What would you tell this CEO to watch out for in working with them?" Even satisfied clients can name friction points. The absence of any friction is a sign of a coached reference.
Ask: "Did they tell you things you didn't want to hear during the engagement? What was that like?" You want a consultant who will push back. References can confirm whether that's real.
Ask: "What did they get wrong, and how did they handle it?" Same principle as asking the vendor directly — you're listening for honesty and recovery, not perfection.
A framework for the decision
After the evaluations are done, I'd suggest a simple framework:
- Trust: do you believe they're telling you the truth, including things that aren't in their interest?
- Relevance: do they have experience with your actual context — industry, company stage, problem type?
- Capability: can they actually build or implement what they're recommending, not just advise on it?
- Alignment: are their incentives aligned with your outcomes, or with billable hours?
Most engaging consultants score well on two or three of these. Firms that score well on all four are rare and worth paying for.
Paul Okhrem advises CEOs on AI strategy and consulting partner selection. More at paul-okhrem.com
Top comments (0)