DEV Community: Sri Ramya

Not Every Test Needs to Be Automated

Sri Ramya — Thu, 08 Oct 2026 16:54:22 +0000

When I started learning about test automation, I used to think: If we can automate a test, why not automate it?

But the more I look at real automation projects, the more I feel that not every test needs to be automated.

A test that we run repeatedly for every release makes sense to automate. But a test that changes often, is used only once, or needs exploratory thinking may not be worth maintaining as automation.

So the question becomes: Can we automate this?

but also: Will this automation give us useful feedback?

While working with X360 AI Tech, One thing we focus on is what happens after the automated test runs.

If a test fails, we don't want the result to stop at a red mark.

We can look at what actually happened during the execution - the browser actions, assertions, and network activity - along with the live execution video, logs, and execution history. This gives us the context around the failure instead of looking at the failed status alone.

Then the failure-analysis part helps us investigate that execution and understand what needs attention.

For example, if a checkout test fails, the important question isn't simply: Did the test fail?

We want to look at where it failed, what happened before the failure, and what the execution evidence is telling us.

That distinction matters because a failed test doesn't automatically mean there is a product bug.

It could be a changed application behaviour, a test issue, or something else in the execution environment.

That's what changed my thinking about automation.

The goal isn't to automate every test.

It's to automate the tests that are worth repeating and when something fails, have enough context to understand what actually happened.

That's where I think automation becomes much more useful.

When Test Automation Scales, What Becomes Harder?

Sri Ramya — Mon, 05 Oct 2026 15:55:22 +0000

When a project has a small number of automated tests, managing them is usually straightforward. But as the application grows, the test suite grows too.

Hundreds or thousands of tests can mean longer execution times, more maintenance, more failures to investigate, and more effort to understand whether the tests are still covering the right things. So I started looking at how different test automation approaches handle this problem.

One thing I noticed is that scaling automation isn't really just about creating more tests.

The bigger challenge is keeping those tests connected to what is actually changing in the application.

For example, when a requirement changes, we may need to understand which tests are affected, which coverage is still relevant, and what needs to be updated before the next execution.

I started looking at X360 AI Tech from this perspective.

What I observe is that it connects different parts of the testing workflow instead of looking at automation as only writing and running tests. Requirements can be brought into the workflow, coverage can be connected to those requirements, tests can be automated and executed, and there are separate areas for self-healing and failure analysis.

That made me think about test automation a little differently.

At a small scale, automation is mainly about saving execution time. At a larger scale, it becomes more about managing change, maintenance, coverage, and failures without losing confidence in the test suite.

For me, that is probably the harder part of scaling test automation. Not having more tests.

Having an automation suite that still makes sense when the product keeps changing.

That’s something I’m still exploring.

When a Code Change Happens, Do We Really Need to Run Every Test?

Sri Ramya — Thu, 01 Oct 2026 14:25:11 +0000

When a small change is made to an application, one thing usually follows:

Run the regression suite.

It makes sense. We want to make sure the change didn't break anything somewhere else. But recently I started thinking about something:

Do we really need to run every test after every code change?

Imagine a test suite with 2,000 tests. A developer makes a change to the payment flow.

Maybe only a part of those tests are directly related to payment. Some others might depend on checkout or orders. And many of the remaining tests may have very little connection to that change. Yet we still run the entire suite.

That's where I started looking into Test Impact Analysis.

The idea is fairly simple: instead of treating every test as equally relevant, can we understand which tests are actually connected to a particular change? That sounds useful, especially when regression suites become large. But I think there is another side to it.

If we run fewer tests, we need to be confident that we didn't skip something important.

This is something I started thinking about more while exploring AI-assisted testing. At X360 AI Tech, I came across an approach where requirements, test coverage, execution, and results are connected instead of treating test execution as an isolated step.

That made me look at regression testing a little differently. It made me think that AI’s value may not always be about creating more tests or running them faster. Maybe it’s also about helping us understand which tests actually matter for the change we just made.

At the same time, I’d want the uncertainty to be visible when AI isn’t confident, rather than silently skipping tests.

For me, the goal isn’t simply: Run fewer tests.

It’s: Run the right tests, and understand why they were selected.

That's something I'm still exploring.

When a Test Passes, What Did It Actually Prove?

Sri Ramya — Mon, 28 Sep 2026 14:35:46 +0000

A passing test usually feels like a good result. The test ran, the assertion passed, and we get a green check. But recently I started thinking about something a little differently:

When a test passes, what did it actually prove?

It proves that the test reached the expected result for that particular scenario. But does it also prove that we tested the right behaviour? Does it prove that we covered the important part of the user flow? And does it tell us what we might still be missing?

That made me look at passing tests a little differently.

I started thinking about this more while exploring AI-assisted testing. As I looked into how AI can help with generating and maintaining tests, A test can pass because everything worked as expected. But it can also pass because the assertion isn't checking enough, the test data doesn't exercise the important case, or the scenario itself doesn't cover the risk we actually care about.

That's where I think the context around a test becomes important. At X360 AI Tech, I came across an approach where requirements, test coverage, execution, and the result are connected instead of looking at the final pass/fail status on its own.

I liked that idea because it changes the question a little.

Instead of only asking: Did the test pass?

we can also ask: What behaviour did this test actually validate?

I don't think this means every passing test needs a detailed investigation. But when AI is generating or maintaining more tests for us, I think this becomes more important.

If creating tests becomes easier, understanding what those tests actually prove may become the harder part. For me, a green test is useful. But knowing "why it is green and what it actually checked" gives me much more confidence in the result.

That's something I'm still exploring.

A Test Can Be Correct and Still Fail for the Wrong Reason

Sri Ramya — Fri, 25 Sep 2026 14:08:10 +0000

A test failing doesn’t always mean there’s something wrong with the test.

Sometimes the test is written correctly, the application is behaving as expected, but something around the test has changed. Maybe the test data was already used, the session wasn’t cleared properly, the environment is in a different state, or a network call took longer than expected. That’s where things get confusing.

When we see a red test, it’s easy to start changing the test itself. But if the real problem was the data or environment, we may end up changing a test that was actually doing its job.

I started thinking about this more while exploring AI-assisted testing. In X360 AI Tech, I came across an approach where the failure is looked at together with the execution context instead of treating the failed test as the problem straight away.

Things like logs, screenshots, traces, test steps, previous results, and application behaviour can help give a clearer picture of what actually happened.

I think this becomes even more important when AI is involved.

If AI sees a failed test and immediately changes something to make it pass, we might get a green result without really understanding what caused the failure in the first place.

For me, the useful question isn’t just: How do we fix this failed test?

It’s: What actually caused this test to fail?

Because sometimes the test isn’t the thing that needs fixing.

Can a Test Be Reliable and Still Be Outdated?

Sri Ramya — Mon, 21 Sep 2026 17:47:23 +0000

I’ve been thinking about something that’s easy to miss with test automation: can a test be completely reliable and still be outdated?

Tools like mabl focus on keeping coverage current as applications change, Testim uses AI-powered locators to make tests more resilient, Functionize looks at AI-driven maintenance and test realignment, and Applitools takes a different angle with visual and functional validation. I like that each approach is solving a different part of the same maintenance problem.

While exploring X360 AI Tech, this made me look at test maintenance a little differently. A test can keep passing every day and still be less useful if the requirement, user flow, or expected behavior has changed. In that case, there’s no failure telling us that something needs attention.

That’s what I find interesting about AI-assisted testing. Maybe AI shouldn’t only help us fix tests when they break. It could also help us notice tests that have been passing for a long time but may no longer represent what we actually want to validate.

I’m starting to think that a healthy test suite isn’t just one with fewer failures. It’s one where we still understand why each important test exists and whether it matches the current product.

How do you usually spot tests that are still passing but have slowly become outdated?

When AI Generates More Tests, What Are We Really Covering?

Sri Ramya — Tue, 15 Sep 2026 17:01:28 +0000

Lately, I’ve been thinking about something that sounds like a good problem to have: AI can help us create a lot more tests, but does having more tests actually mean we have better coverage?

There are already some interesting approaches. Katalon can generate test cases from requirements, ACCELQ looks at more autonomous test generation, LambdaTest’s KaneAI works from natural-language scenarios, and Functionize can turn test instructions into executable tests. Each approach makes creating tests easier, but I think the harder part starts after the generation.

While exploring X360 AI Tech, I started looking at this from another angle. Instead of asking how many tests AI can generate, I’m more interested in whether those tests actually cover the important parts of the requirement - the main user flow, edge cases, business rules, and things that could easily be missed.

Because AI can generate 20 tests where maybe only 5 are really useful. And it can also miss one important scenario while giving us the feeling that we have good coverage.

I’m starting to think that the number of AI-generated tests is probably not the best measure of coverage. I’d rather have fewer tests that cover the right risks than a huge suite full of similar scenarios.

For me, that’s where QA still matters. The question isn’t just “How many tests did AI create?” but “What did those tests actually cover?”

An AI-Fixed Test Passed. What Should QA Check Next?

Sri Ramya — Tue, 08 Sep 2026 15:40:37 +0000

One thing I’ve been thinking about something that sounds simple but is actually a little tricky: what do we do after AI fixes a failed test and it passes again?

There are already some interesting approaches to this. mabl looks at adaptive healing, Testim focuses on smarter locators and maintenance, and Applitools approaches changes from the visual validation side. I don’t think there’s one perfect way to handle test maintenance, it really depends on why the test failed in the first place.

While exploring X360 AI Tech, this made me look at the problem a little differently. Getting a test back to green is useful, but I’m more interested in what happened along the way. Looking at the failure details, previous execution, and the actual flow can help answer a basic question: did we really fix the test, or did we just find another way to make it pass?

So, after an AI fix, I’d still want to check a few things: Is the test checking the same thing as before? Does it still match the original requirement? Was the failure actually caused by a UI change? And does the fix continue to work in the next few runs?

I’m starting to feel that the real value of AI self-healing isn’t just fixing tests faster. It’s helping QA spend less time fixing tests blindly and more time deciding whether the fix actually makes sense.

What’s the first thing you would check after an AI-healed test turns green?

When a Test Passes, How Do We Know We Can Trust It?

Sri Ramya — Fri, 04 Sep 2026 15:01:41 +0000

A green test result usually feels like a good sign. But I’ve started wondering if a passing test always means everything went right. Sometimes a retry makes it pass, an assertion may not be checking enough, or the test may pass without really validating what we wanted. So the question I keep coming back to is: why do we trust that it passed?

While exploring tools like mabl, Testim, and Applitools, I noticed they look at this from different sides - execution history and evidence, step-level details, or visual validation. While looking at X360 AI Tech as well, I found myself paying more attention to execution history and video evidence, because the final PASS alone doesn’t really tell the whole story.

For me, a passing test makes more sense when I can see what actually happened - did the right scenario run, was the expected outcome checked, did it pass on the first try, and is there enough evidence to understand the result? This becomes even more important when AI is creating tests, changing locators, or adapting to UI changes.

Curious to know how others think about what they usually check before trusting a passing automated test?

When AI Fixes a Test, How Do We Know It Fixed the Right Thing?

Sri Ramya — Tue, 01 Sep 2026 14:25:13 +0000

While exploring AI-assisted testing, I started thinking more about what actually happens when a test breaks. Tools like mabl, Testim, and Functionize are approaching self-healing in different ways, and I can see why it’s becoming useful as test suites get bigger.

But I keep coming back to one thing: just because a healed test passes again, it doesn’t always mean it was fixed correctly. If a locator breaks and AI finds another element that makes the test pass, we still need to know whether it’s actually the element the test was meant to use.

While exploring these ideas through X360 AI Tech, I’ve also been looking at how much context is needed for a test change to actually make sense. For me, it’s not just about finding something that works again, but making sure the fix still supports the original purpose of the test. When that isn’t clear, I think QA review is still important.

There’s a big difference between “the test passed again” and “the test passed again, and we know it was fixed correctly.” I think that difference becomes more important as test suites grow.

When an Automated Test Fails, I’ve Started Looking Beyond the Failure

Sri Ramya — Sat, 29 Aug 2026 06:41:34 +0000

After exploring AI-generated testing, I started paying more attention to what happens when those tests actually run and something fails.

A failed test doesn’t always mean there’s a bug in the application. Sometimes a locator changed, sometimes the test is outdated, and sometimes there’s a real issue with the product.

While looking at different approaches, I found mabl interesting for its focus on adaptive test maintenance and failure analysis. Testim also takes an AI-assisted approach to keeping tests resilient as applications change. Functionize goes further into self-healing and helping identify the root cause behind failures.

What I found interesting while exploring X360 AI Tech was looking at test maintenance as part of a bigger flow - from the requirement and automation to execution and understanding what actually went wrong. It made me look at self-healing a little differently.

Fixing a failed test automatically can definitely save time. But if we don’t understand why the test failed, we might just be hiding the actual problem.

For me, the useful part of AI-assisted maintenance is not simply making a failed test green. It’s helping us understand what changed, giving enough context around the failure, and then deciding whether it’s something safe to update or something that needs a QA to look at.

I’m starting to see AI test maintenance less as “AI fixes broken tests” and more as “AI helps us understand what actually needs fixing.” That feels like a much more practical use of AI in testing.

How I’m Looking at AI-Generated Tests Differently

Sri Ramya — Wed, 26 Aug 2026 14:04:36 +0000

While exploring different AI-powered testing tools, I started noticing that generating a large number of test cases doesn’t necessarily mean we’re getting better tests.

For example, Katalon uses AI to help generate tests from requirements, while Tricentis Tosca takes a more model-based approach to creating and maintaining test assets. Applitools is more focused on using AI for visual validation. Each one seems to solve a different part of the testing process.

I found those differences interesting, and while exploring X360 AI Tech, I started looking at AI-generated testing from another angle - how well the requirement, test scenarios, and automation actually connect with each other.

That made me realize that a good AI-generated test isn't necessarily the one with the most steps or the most scenarios. What matters more is whether it covers something that actually needs to be tested.

The same applies to self-healing. A test being fixed automatically and turning green is useful, but only if it’s still checking what it was originally meant to check.

So lately, I’ve been paying less attention to how many tests AI can generate and more attention to whether those tests actually represent the requirement and provide meaningful coverage.

That shift in perspective has been one of the more interesting things I’ve noticed while exploring AI-assisted testing.