I have done this process wrong more times than I would like to admit.
You call the references the vendor sends you. Three customers, all happy, all articulate, all saying roughly the same things. You hang up feeling good. You sign. Six months later you are dealing with a support team that responds every 72 hours and a renewal quote that is 40% higher than year one.
The reference check told you nothing useful. Not because the customers lied. Because you asked the wrong questions.
Here is what I ask now.
"How many people at the vendor have you spoken to in the last six months?"
One contact who handles everything and is sometimes slow — that tells you something. A team of people across sales, technical support, and leadership — that tells you something different. Enterprise AI vendors with thin account teams show their limitations at exactly the moment you need them most: when something breaks at the worst possible time.
"Tell me about the last time something broke in production."
Not IF something broke. Something always breaks. I want to know what happened next. Did the vendor show up? Did they communicate clearly while the issue was live? Did they follow up after closing the ticket, or did they close it and disappear?
The answer to this question tells me more about a vendor than any product demo.
"What do you know now that you wish you had known before signing?"
This question works because it is framed as advice, not criticism. Reference customers who would never say "this product has problems" will happily answer this one honestly. Listen for anything about pricing surprises, scope limitations that only appeared after deployment, or feature gaps the sales team glossed over.
"When you renewed, did you evaluate alternatives?"
Renewal is the honest signal. A customer who renewed without looking elsewhere is genuinely satisfied. A customer who looked at three other options and came back is someone who chose this vendor over real competition. A customer who is approaching renewal and is not sure is the most valuable reference call you can make — if you can get the vendor to put them on the list, which they will not.
"How does it work for someone who joined the team after the initial rollout?"
The initial deployment had vendor support, internal champions, and everyone paying close attention. The ongoing experience for employees who joined later and had to learn the tool on their own is the real daily experience. This is where most enterprise AI tools quietly underperform their pilot results.
None of these questions are aggressive. A vendor whose customers have good experiences will answer them confidently. A vendor whose customers have mixed experiences will give you information that changes what you negotiate before signing.
Both outcomes are better than what you get from asking "do you like the product?"
Top comments (0)