Why Flaky Tests Are More Dangerous Than Failed Tests

Hari Gokul — Thu, 24 Sep 2026 09:52:29 +0000

A failed test is usually easy to notice.

It turns red, someone checks it, finds the problem, fixes it, and moves on.

Flaky tests are different. They pass sometimes and fail sometimes, and the difficult part is figuring out why. After seeing the same test fail randomly several times, people can slowly start treating it as "just another flaky test."

That's where things become risky.

Flaky tests can come from timing issues, unstable selectors, network delays, test data, asynchronous operations, or differences between environments. Sometimes the application is working perfectly and the test is the problem. Other times, the failure is actually pointing to a real bug.

The biggest problem is that people eventually stop trusting the test results.

Imagine a CI pipeline with hundreds of tests. If a few tests are known to fail randomly, developers may simply rerun the pipeline. But what if one of those failures is actually a real issue?

This is why we started looking at flaky tests differently while working on X360aitech. The goal isn't just to make the pipeline green. It's to understand what caused the failure and give testers enough information to decide what to do next.

Playwright is great for browser automation, but the framework itself can't solve every source of flakiness. Test design, stable locators, proper waits, isolated test data, and a reliable environment are still important. In X360aitech, we look at these execution details together with AI-assisted analysis rather than treating a failed test as just a red status.

This is where AI becomes interesting. With X360aitech, execution information such as logs, screenshots, traces, and previous test results can be used to help identify patterns around failures. Instead of spending time going through everything manually, testers can get a better starting point for investigation.

Self-healing is another area we're exploring with X360aitech. If a UI change breaks a selector, AI can help identify whether the element has changed and whether a possible replacement actually makes sense.

But there's an important distinction: self-healing shouldn't mean making every failed test pass.

If a test fails three times and passes on the fourth attempt, that doesn't automatically mean the problem is solved. The important question is still: why did it fail?

That's also how we're thinking about AI-assisted testing in X360aitech. AI should help with investigation and maintenance, but it shouldn't hide failures or make changes without enough context.

For me, good test automation isn't about having the biggest test suite. It's about having tests that the team can actually trust.

I'd rather have a smaller set of stable and meaningful tests than thousands of tests that everyone ignores when they fail.

So the next time a test fails randomly, instead of immediately clicking "Rerun", maybe ask:

Why did it fail?

Is it really flaky?

Or is it telling us something important about the application?

That's the kind of problem we're trying to solve with X360aitech making automated testing not just faster, but easier to understand and maintain.

How does your team deal with flaky tests — fix them immediately, quarantine them, or just rerun them?

Why AI Is Changing the Way We Think About Test Automation

Hari Gokul — Tue, 22 Sep 2026 16:07:24 +0000

Test automation has made testing a lot easier. Tools like Selenium and Playwright help us automate repetitive tasks and save a lot of time. But anyone who has worked with automation for a while knows that writing the test is only the beginning.

As the application grows, tests also become harder to maintain. A small UI change can break a selector, a test can become flaky, or a failure can take a lot of time to understand. Sometimes the application is perfectly fine and the problem is actually with the test.

This is one of the areas where I think AI can be genuinely useful in testing.

Instead of using AI only to generate test scripts, we can use it to understand requirements, think about different test scenarios, analyze failures, and help maintain existing tests.

For example, if we have a requirement for a login feature, AI can help think beyond the normal successful login. It can suggest cases for invalid credentials, empty fields, locked accounts, and other edge cases. A tester still needs to review those scenarios, but it can be a good starting point and save some manual effort.

AI can also help when it comes to automation itself. A tester can describe what they want to test in natural language, and AI can help translate that intent into automation. This is one of the ideas we are exploring with X360aitech, where AI-assisted testing works alongside technologies like Playwright.

The idea isn't to replace Playwright or other automation frameworks. Playwright can handle the actual browser execution while AI helps with understanding the testing intent and the context around it.

Test maintenance is another interesting area. When the UI changes, an existing test might stop working even though the feature itself is still working. AI-assisted self-healing can help identify these changes and suggest a way to recover the test.

But this needs to be done carefully. Making a test pass isn't the same as fixing a test. If AI picks the wrong element and the test passes, we might actually be hiding a problem instead of solving one.

The same applies to failed tests. When something fails, testers often have to go through logs, screenshots, traces, and previous results to understand what happened. AI can help bring this information together and give us a better starting point for the investigation. This is another area we are exploring with X360aitech.

For me, the interesting part isn't simply generating more tests. It's making the whole testing process a little smarter understanding what needs to be tested, creating the automation, running it, understanding failures, and improving the tests over time.

And I don't think AI means testers become less important. Someone still needs to understand the product, question the results, and decide whether something is actually a bug. AI can take care of some repetitive work, but the final thinking still needs to come from people.

That's the direction we're exploring with X360aitech combining AI with automation and human judgment to make testing easier to create, maintain, and understand.

DEV Community: Hari Gokul

Why Flaky Tests Are More Dangerous Than Failed Tests

Why AI Is Changing the Way We Think About Test Automation