Every AI vendor has a demo that works perfectly. That is the problem.
Gartner predicts over 40% of agentic AI projects will be canceled by the end...
For further actions, you may consider blocking this person and/or reporting abuse
red flag 2 burned me harder than any other. we approved budget after a demo on their curated dataset, then spent two months realizing our actual data killed accuracy by 40%. vendor response was basically "your data needs cleanup" which... sure, but that gap should have been the first conversation not the last.
the sales engineer test in flag 1 is genuinely underrated. best filter question i've found is "what breaks first?" - if they don't have a confident fast answer, they haven't run it at scale.
That "your data needs cleanup" deflection is basically the vendor admitting they knew the gap existed and chose not to surface it. A honest demo would have asked for a sample of your actual data before the budget conversation even started.
the 'your data needs cleanup' line is almost a red flag in itself at this point - i've heard it enough times to recognize it as a pattern. the better vendors ask for a dirty sample upfront specifically to show how they handle it. the ones who don't usually already know the answer won't look good
Red flag 3 (no failure mode discussion) is the one I see trip up teams the most. I'd add a practical test: ask the vendor to run the demo with deliberately bad input — malformed data, edge cases, adversarial prompts. If they can't or won't, that tells you everything about how far the product is from production-ready. The best AI tools I've worked with fail gracefully and tell you why they failed. The worst ones just confidently return wrong answers.
Deliberately bad input is the best vendor filter — if they flinch at malformed data during a demo, imagine what happens in production with real user chaos. That "run it with garbage" test should honestly be step one in every PM's eval checklist.
Red flag 4 (hidden costs at scale) is the one I wish more teams talked about openly. I run a system that makes thousands of automated calls across multiple APIs and LLMs daily — the difference between prototype costs and production costs isn't linear, it's exponential once you factor in retries, error handling, and the data normalization layer you inevitably have to build.
One thing I'd add to the cheat sheet: ask about degradation behavior. Not just "what happens when it's wrong" but "what happens when your upstream model provider has a bad day?" I've seen agents that work flawlessly on Claude 3.5 suddenly produce garbage when the provider silently updates model weights or changes rate limits. The best production systems I've built have fallback chains and output validation layers — and that's infrastructure the vendor demo will never show you.
The "ask the sales engineer, not the AE" advice is gold. I'd extend it further: ask to talk to their on-call engineer. That person knows every failure mode the product has, because they've been woken up by each one.
Retries are the silent budget killer — we saw one pipeline where retry loops alone 3x'd our token spend before anyone noticed. Logging cost-per-completed-action instead of cost-per-request changed how we think about scaling entirely.
Cost-per-completed-action is such a better mental model than cost-per-request. I've seen the same pattern with automated pipelines — you don't realize how much retries are costing until you actually instrument at the action level.
Curious: did you end up adding circuit breakers or just capping retry attempts? I found that exponential backoff with a hard ceiling on total retries per action was the sweet spot for keeping costs predictable without sacrificing reliability.
christ does anyone on this entire website write anything themselves?
Fair challenge — I write from building AI products daily and getting burned by exactly these demo tricks. Happy to debate any specific point if something felt off.