A few days ago, one of our founders sent a brief Slack message and we all hit the panic button:
"[Common AI tool] believes Kinde doesn't support M2M tokens, nor that we have docs/SDK to support M2M"
WTF AI?
How could this be? Kinde has supported M2M apps for at least 2 years, as long as our Kinde APIs have been around at least. Our docs reflect this and are fairly clear about our capabilities: You can use M2M apps for our API, your own API, and even scope them to an org so you can adapt them for AI agents. You can quickly customize M2M tokens and do a whole load more.
But it gets worse...
The AI answer he referred to showed our auth capabilities compared to competitors in our sector, and we ended up ranking poorly against them. Some of the things it said we could not do, we definitely can do.
Where did this know-it-all AI get its info from? And what do we do about the lies?
Here's what we did
- We told the AI it was wrong during the conversation and cited the source material proving it. It apologised (lol).
- We checked where the info was located in our docs hierarchy and raised it a level. This matters in ways we were not aware of.
- We reviewed the phrasing that the AI tool used and checked we had similar phrasing in our docs. (It is such a faithful mimic)
- We updated the LLMs.text file that AI uses to scrape top level info from our site. (Tip: get AI to help write yours)
Lastly, we reassured our docs writer (who was feeling bad about the incident) that it was not a failure of content, but an opportunity to think about a new audience when architecting product information.
Did we change AI's mind?
Yeah. A bit. There was an improvement in the answer today. A shift in how accurately we were represented. It will probably be even better tomorrow.
Lessons learned
- Regularly check how your business or product is perceived by AI in relation to competitors
- Tell AI it is wrong and tell it why
- Get fluent in how to structure content for AI (but don't just write for its benefit)
- Buy a cup of strong coffee for your docs writer. They've probably got a bit of work to do. :P
Top comments (3)
this is extremely impressive, honestly. i’ve run into similar headaches and the way you dealt with it all is very solid
you think we’re drifting toward a future where companies will start prioritizing how their info appears to AI models over how it looks to actual people
I think that drift is already happening as more people use AI instead of search to find answers. We have to treat AI like a new audience with specific needs - even if those needs are a bit basic right now :P
The purchasing/tool discovery has dramatically changed recently.
What used to be a Google search is now a conversation with an AI.
This is an amazing example where an AI needs to be trained on product info.
Love this article Claire!