I have extensive experience in testing, automation, and setup of all these systems, and have successfully led numerous projects in the past that have fixed and scaled up automation, even for non-technical engineers.
Participated and consulted with around ten companies about how to fix their Agentic testing or automation approach.
In most cases, automation and testing were so unreliable that they even resulted in false positives, prompting complaints from C-level executives, managers, and engineers about their decision to shift Test Engineers or Automation engineers.
Some of the Agentic testing providers claim to develop around 1000 test cases monthly, but after a few months, they deliver around 500 largely flaky tests.
Some providers have claimed zero flakiness, connect Confluence or documents, and our tool will build excellent test coverage for every feature without requiring human interaction.
I had a chance to listen to their demos, watch their ads, and websites with excellent descriptions
But most of their demos, sites, and ads are based on just login, or just fulfilling a simple user registration setup
But moreover, I see lots of articles on the engineering media that "Playwright MCP will blow your mind" and it's the same - login scenario, maximum simple registration, and shitty code generated based
So, does anybody have a success story for a big product to shift human Test Engineering to Agentic testing tools?
Top comments (2)
I’ve seen lots of “agentic testing” tools make big promises, but flaky tests and false positives seem to be a common problem.
In your experience, what makes the difference between a successful automation setup vs one that degrades over time?
Demos cover only simple flows; complex scenarios break everything.