Last month a customer emailed me furious, asking why their full refund had not hit their bank account. Our policy has always been 14 days strict.
Then they sent me a screenshot of their chat with our AI support bot.
Around turn seven of the conversation, the bot casually promised them an unconditional 90 day money back guarantee. After the recent Air Canada court ruling, companies are legally on the hook for whatever their bot promises, so I had to eat the cost and pay them.
I was baffled because the bot passed every single prompt test I ran in the playground.
What I did not realize was the brutal compounding math of multi-turn conversations. A 95% accuracy score on single prompts drops to 77% by turn 5, a coin flip by turn 10, and barely 35% by turn 20. As the chat gets longer, conversation tokens push your system prompt constraints out of attention focus, and the bot starts hallucinating rules to please the user.
I got tired of flying blind and writing messy Python scripts every time I changed a prompt, so I built QAgent (https://qagent.in).
It connects to your bot with a simple webhook and runs automated multi-turn test suites across policy rules, RAG accuracy, and context memory, showing you the exact turn where your agent drifts before real users see it.
There is a free tier with 100 tests a month without any credit card at qagent.in so other builders can avoid eating refund costs like I did.
If you have an agent live right now, check your multi-turn chat logs. You might be surprised what it promises when conversations go past turn five.
Top comments (0)