DEV Community: Sri Ramya

Can AI Know When It Doesn't Have Enough Information to Test a Feature?

Sri Ramya — Thu, 29 Oct 2026 16:31:19 +0000

Recently, I started thinking about something while exploring AI in software testing.

AI can generate test cases from requirements in seconds. But what if the requirement itself is not clear enough?

Let's take a simple example.

A requirement says, "Lock the user account after multiple failed login attempts."

Now, how many failed attempts should be allowed? How long should the account stay locked? Can the user unlock it through email?

If the requirement doesn't explain these details, how does AI know what the correct behaviour should be?

It might make its own assumptions and generate test cases based on them. The test cases may look good, but they could still be testing the wrong behaviour. And that's what got me thinking.

Sometimes, the problem isn't how AI generates test cases. It's whether AI has enough information to generate the right ones.

I started looking at how AI-assisted testing handles unclear requirements.

Generating test cases is useful, but identifying missing information is equally important.

For example, instead of assuming how many failed login attempts are allowed, AI could point out that this detail is missing and ask the QA engineer to clarify it.

This gives the team a chance to confirm the expected behaviour before testing begins. Otherwise, we might end up with test cases that look complete but are based on assumptions nobody actually agreed on.

While exploring X360 AI Tech, I looked at how requirements can be connected to test case generation.

One thing that stood out to me is the importance of understanding requirements before turning them into tests. x360 AI Tech provides AI-assisted workflows for working with requirements and generating test cases, with an AI assistant that can ask follow-up questions.

This made me think about how useful it is to identify missing details early instead of discovering them only after test cases have been created.

Of course, AI cannot decide every business rule on its own. The QA engineer and team still need to confirm what the application is supposed to do.

But if AI can help us spot unclear requirements earlier, it can save time and help us create tests that actually match the expected behaviour.

For me, good AI testing is not just about generating more test cases. It's about asking the right questions before generating them.

When a requirement is unclear, would you prefer AI to stop and ask questions first, or generate test cases with its assumptions clearly marked?

I'd love to hear how other QA engineers handle this in their projects.

How Can Smart Test Selection Reduce Regression Risk?

Sri Ramya — Mon, 26 Oct 2026 15:24:33 +0000

Whenever developers change code, QA teams need to check whether the change has affected existing functionality.

One common approach is to run the entire regression test suite. This helps identify unexpected issues, but as the number of tests grows, execution can take more time.

Running only a few tests may save time, but it can also leave important bugs undetected. So, how can we reduce regression testing time without increasing the risk of missing bugs?

Let's understand with a simple example

Imagine a banking application where users can transfer money between accounts. A developer changes the logic that calculates transfer fees.

The QA team needs to check whether the change affects:

The transfer fee calculation
The total amount deducted from the sender's account
The amount received by the other account
The transaction history

But should the team also rerun every test related to login, profile settings, and notifications?

The answer depends on how these features are connected to the changed code. Running every test after every small change can waste time. But testing only the modified function might miss problems in other connected areas.

Instead of running the entire test suite every time, teams can use code changes and impact analysis to identify tests that are relevant to the change.

A practical approach includes:

Identify what was modified and which application components might be affected. Use code relationships, test mappings, and impact analysis to find tests associated with the changed areas.

Even if a test isn't directly linked to the modified code, check whether it covers an important business operation, such as transferring money or verifying transaction records.

Run the selected tests and investigate failures using available execution evidence. If dependencies are unclear or a critical flow might be affected, running additional regression tests may be the safer choice.

The goal isn't simply to run fewer tests. It's to make informed decisions about which tests to run and when.

While exploring x360 AI Tech, I found its Smart Test Selection approach relevant to this challenge.

The platform describes selecting relevant tests based on code changes and impact analysis. This can help teams focus their testing efforts on potentially affected areas rather than always running the entire test suite.

However, test selection cannot guarantee that every affected scenario will be identified. Indirect dependencies and critical business flows still need careful consideration.

For me, the important part is balancing execution speed with confidence in the release. Saving testing time is useful only when the team has a sensible way to manage the risk of missing a defect.

When you reduce regression testing after a code change, how do you make sure you're not skipping a test that could catch a critical bug?

A Requirement Changed. How Do We Know Which Tests to Revisit?

Sri Ramya — Wed, 21 Oct 2026 11:30:43 +0000

Requirements rarely stay the same throughout a project.

A new business rule gets added, an existing feature changes, or a customer asks for a different behaviour. The requirement is updated, the code changes, and the team moves forward.

But there's one question that can easily get overlooked:

When a requirement changes, how do we know which existing tests need to change or run again?

Imagine an online shopping application where customers can request a refund within 30 days of delivery.

Now, the business changes that rule to 14 days.

Updating the refund test case is the obvious step. But what about tests for refund eligibility, order history, payment status, or related checkout flows? Could the changed rule affect any of them?

The challenge isn't just updating one test. It's understanding the possible impact of that change across the application.

If the team reruns the entire regression suite after every small change, feedback can become slower. But if they run only the test they already know is affected, they could miss an unexpected issue elsewhere.

This is where change impact analysis becomes important.

While researching how teams approach this problem, I found that maintaining clear links between requirements, test cases, coverage, and execution results can make impact analysis more manageable. Instead of relying only on memory or manually searching through a large test suite, teams can use those relationships to identify which tests deserve review.

While exploring x360 AI Tech, I found its Smart Test Selection approach particularly relevant to this problem. The platform describes selecting relevant tests based on code changes and impact analysis, alongside connecting requirements with test coverage.

That connection is useful because teams don't always need to rerun every test after every change. They need a reliable way to decide what deserves attention first. With requirement-linked coverage, relevant test selection, and execution evidence, QA can investigate the change more systematically instead of relying only on manual searches.

The important part is still validating the selection. A requirement change can affect behaviour beyond the directly modified code, so QA needs to review the business impact and decide whether broader regression testing is necessary.
A requirement change shouldn't automatically mean rerunning everything. But it shouldn't mean updating just one test and assuming the rest are safe either.

I want to know that when a requirement changes in your project, how does your team identify the tests that need to be updated or rerun—through requirement-to-test traceability, impact analysis, automation, or mostly manual review?

When AI Agents Test Software, Who Checks the Tester?

Sri Ramya — Mon, 19 Oct 2026 15:17:38 +0000

AI is becoming capable of doing more than generating test cases. AI agents can help plan test steps, interact with applications, execute workflows, and analyze failures.

That sounds promising. But while exploring AI-assisted testing, I kept coming back to one question:

If AI can test an application, how do we know it tested the right thing?

Let's take a simple example.

Imagine an AI agent testing an online shopping website. It selects a product, clicks Checkout, and completes the payment flow. The test reports a pass.

But was the payment actually successful? Was the correct order created, and was the right amount charged?

That's where QA comes in. The QA engineer needs to verify whether the AI tested the correct business rules, checked the expected results, and missed any important scenarios.

The test passing is one thing. Knowing that it tested the right thing is another.

The test executed successfully, but did it really verify that the application worked correctly?

That's the difference I've been thinking about. A test passing doesn't automatically mean the right behaviour was tested.

This becomes more important as AI agents take on more testing responsibilities. Generating a test, executing it, and evaluating the result are different steps. We still need to understand whether the test matches the original requirement, whether its assertions are meaningful, and whether important scenarios have been covered.

While exploring different approaches to AI-assisted testing, I found it useful to look beyond test generation alone. The wider workflow matters too: understanding requirements, identifying coverage, executing tests, maintaining them when the application changes, and investigating failures with useful evidence.

This is what makes me think about the broader testing workflow at X360 AI Tech. The focus connects requirements, test coverage, automation, self-healing, and failure analysis rather than treating each as an isolated activity.

For example, execution evidence such as logs, screenshots, videos, and network activity can help QA understand what happened during a test instead of relying only on a pass-or-fail status. That context can make investigation easier, although QA still needs to verify whether the test and its conclusions are correct.

I don't think the goal should be to remove QA from the process. It should be to reduce repetitive effort while giving QA better context to make informed decisions.

As AI agents become more involved in software testing, I think checking the quality of their work will become just as important as measuring how much they can automate.

I'm curious to know if an AI agent creates, executes, and analyses a test, what would you want to verify before trusting its result?

Your Test Suite Is Growing. But Is It Getting Better?

Sri Ramya — Thu, 15 Oct 2026 17:01:52 +0000

When a project starts, maintaining automated tests usually feels manageable.

There are a few test cases, execution is quick, and when something fails, finding the reason isn't too difficult.

But as the application grows, things start to change.

More features mean more tests. More tests mean longer execution times. And as the UI and application behaviour change, some existing tests need regular maintenance.

Before long, the team might be dealing with hundreds of tests, repeated failures, and a test suite that takes much longer to run than it used to.

The problem isn't that the test suite is growing. It's whether the team can still trust it.

Imagine running 600 tests and getting 30 failures.

Some might be real bugs. Others could be broken locators, outdated test data, environment issues, or flaky tests.

Now the team has another problem: figuring out which failures actually need attention.

This is something I've been exploring at X360 AI Tech.

Instead of looking at test results as just pass or fail, we're interested in how execution details can help make failures easier to investigate. Test steps, assertions, browser actions, network activity, logs, screenshots, and execution videos can provide useful context about what happened.

The goal isn't simply to add more tests or make every failure disappear.

It's to help the team understand what happened during execution and decide what needs attention.

I also think growing test suites need regular review.

Some tests protect critical business flows and should remain part of the regression suite. Others may be redundant, unstable, or no longer relevant after application changes.

More tests don't automatically mean better coverage. And better coverage doesn't automatically mean better feedback.

For me, a healthy test suite is one that covers important behaviour, runs reliably, and helps developers understand problems without spending hours investigating every failure.

As a test suite grows, maintaining trust in the results becomes just as important as adding new tests.

I'm curious how other teams handle this.

When your test suite starts getting bigger and slower, what do you focus on first: reducing execution time, fixing flaky tests, or removing tests that no longer add much value?

Writing the test is 10% of its life. Maintaining it is the other 90%.

Sri Ramya — Tue, 13 Oct 2026 18:08:42 +0000

When I started exploring test automation, I used to think the hardest part was writing the test.

But the more I look at real testing workflows, the more I realize that writing the test is only the beginning. Keeping it reliable as the application changes is where much of the effort goes.

Imagine running 500 automated tests overnight and finding 23 failures the next morning.

The difficult part isn't finding out which tests failed. It's understanding why.

Is there an actual bug?
Did a UI change break a locator?
Is the API or test environment causing the issue?
Is the test using incorrect data, or is it simply flaky?

Each failure needs investigation, and when the test suite grows, this can take more time than expected.

This is something I've been exploring at X360 AI Tech.

Instead of looking at a failed status alone, we're looking at how execution details can help with failure analysis. Test steps, assertions, browser actions, network activity, logs, screenshots and execution videos can provide useful context about what happened during a run.

The goal isn't to assume that AI can identify every failure correctly. It's to help engineers start their investigation with more information instead of searching through everything from scratch.

For example, if a checkout test fails, knowing the exact step where it failed and what happened just before it can make it easier to investigate whether the problem is in the application, the test, or the environment.

I think this is an important part of test maintenance that doesn't always get enough attention.

A test suite isn't valuable just because it has hundreds of tests. It's valuable when those tests remain reliable and give the team feedback they can trust.

I'm curious how other teams approach this.

When a test starts failing repeatedly after application changes, do you first investigate and repair it, or do you sometimes question whether that test is still worth maintaining?

Not Every Test Needs to Be Automated

Sri Ramya — Thu, 08 Oct 2026 16:54:22 +0000

When I started learning about test automation, I used to think: If we can automate a test, why not automate it?

But the more I look at real automation projects, the more I feel that not every test needs to be automated.

A test that we run repeatedly for every release makes sense to automate. But a test that changes often, is used only once, or needs exploratory thinking may not be worth maintaining as automation.

So the question becomes: Can we automate this?

but also: Will this automation give us useful feedback?

While working with X360 AI Tech, One thing we focus on is what happens after the automated test runs.

If a test fails, we don't want the result to stop at a red mark.

We can look at what actually happened during the execution - the browser actions, assertions, and network activity - along with the live execution video, logs, and execution history. This gives us the context around the failure instead of looking at the failed status alone.

Then the failure-analysis part helps us investigate that execution and understand what needs attention.

For example, if a checkout test fails, the important question isn't simply: Did the test fail?

We want to look at where it failed, what happened before the failure, and what the execution evidence is telling us.

That distinction matters because a failed test doesn't automatically mean there is a product bug.

It could be a changed application behaviour, a test issue, or something else in the execution environment.

That's what changed my thinking about automation.

The goal isn't to automate every test.

It's to automate the tests that are worth repeating and when something fails, have enough context to understand what actually happened.

That's where I think automation becomes much more useful.

When Test Automation Scales, What Becomes Harder?

Sri Ramya — Mon, 05 Oct 2026 15:55:22 +0000

When a project has a small number of automated tests, managing them is usually straightforward. But as the application grows, the test suite grows too.

Hundreds or thousands of tests can mean longer execution times, more maintenance, more failures to investigate, and more effort to understand whether the tests are still covering the right things. So I started looking at how different test automation approaches handle this problem.

One thing I noticed is that scaling automation isn't really just about creating more tests.

The bigger challenge is keeping those tests connected to what is actually changing in the application.

For example, when a requirement changes, we may need to understand which tests are affected, which coverage is still relevant, and what needs to be updated before the next execution.

I started looking at X360 AI Tech from this perspective.

What I observe is that it connects different parts of the testing workflow instead of looking at automation as only writing and running tests. Requirements can be brought into the workflow, coverage can be connected to those requirements, tests can be automated and executed, and there are separate areas for self-healing and failure analysis.

That made me think about test automation a little differently.

At a small scale, automation is mainly about saving execution time. At a larger scale, it becomes more about managing change, maintenance, coverage, and failures without losing confidence in the test suite.

For me, that is probably the harder part of scaling test automation. Not having more tests.

Having an automation suite that still makes sense when the product keeps changing.

That’s something I’m still exploring.

When a Code Change Happens, Do We Really Need to Run Every Test?

Sri Ramya — Thu, 01 Oct 2026 14:25:11 +0000

When a small change is made to an application, one thing usually follows:

Run the regression suite.

It makes sense. We want to make sure the change didn't break anything somewhere else. But recently I started thinking about something:

Do we really need to run every test after every code change?

Imagine a test suite with 2,000 tests. A developer makes a change to the payment flow.

Maybe only a part of those tests are directly related to payment. Some others might depend on checkout or orders. And many of the remaining tests may have very little connection to that change. Yet we still run the entire suite.

That's where I started looking into Test Impact Analysis.

The idea is fairly simple: instead of treating every test as equally relevant, can we understand which tests are actually connected to a particular change? That sounds useful, especially when regression suites become large. But I think there is another side to it.

If we run fewer tests, we need to be confident that we didn't skip something important.

This is something I started thinking about more while exploring AI-assisted testing. At X360 AI Tech, I came across an approach where requirements, test coverage, execution, and results are connected instead of treating test execution as an isolated step.

That made me look at regression testing a little differently. It made me think that AI’s value may not always be about creating more tests or running them faster. Maybe it’s also about helping us understand which tests actually matter for the change we just made.

At the same time, I’d want the uncertainty to be visible when AI isn’t confident, rather than silently skipping tests.

For me, the goal isn’t simply: Run fewer tests.

It’s: Run the right tests, and understand why they were selected.

That's something I'm still exploring.

When a Test Passes, What Did It Actually Prove?

Sri Ramya — Mon, 28 Sep 2026 14:35:46 +0000

A passing test usually feels like a good result. The test ran, the assertion passed, and we get a green check. But recently I started thinking about something a little differently:

When a test passes, what did it actually prove?

It proves that the test reached the expected result for that particular scenario. But does it also prove that we tested the right behaviour? Does it prove that we covered the important part of the user flow? And does it tell us what we might still be missing?

That made me look at passing tests a little differently.

I started thinking about this more while exploring AI-assisted testing. As I looked into how AI can help with generating and maintaining tests, A test can pass because everything worked as expected. But it can also pass because the assertion isn't checking enough, the test data doesn't exercise the important case, or the scenario itself doesn't cover the risk we actually care about.

That's where I think the context around a test becomes important. At X360 AI Tech, I came across an approach where requirements, test coverage, execution, and the result are connected instead of looking at the final pass/fail status on its own.

I liked that idea because it changes the question a little.

Instead of only asking: Did the test pass?

we can also ask: What behaviour did this test actually validate?

I don't think this means every passing test needs a detailed investigation. But when AI is generating or maintaining more tests for us, I think this becomes more important.

If creating tests becomes easier, understanding what those tests actually prove may become the harder part. For me, a green test is useful. But knowing "why it is green and what it actually checked" gives me much more confidence in the result.

That's something I'm still exploring.

A Test Can Be Correct and Still Fail for the Wrong Reason

Sri Ramya — Fri, 25 Sep 2026 14:08:10 +0000

A test failing doesn’t always mean there’s something wrong with the test.

Sometimes the test is written correctly, the application is behaving as expected, but something around the test has changed. Maybe the test data was already used, the session wasn’t cleared properly, the environment is in a different state, or a network call took longer than expected. That’s where things get confusing.

When we see a red test, it’s easy to start changing the test itself. But if the real problem was the data or environment, we may end up changing a test that was actually doing its job.

I started thinking about this more while exploring AI-assisted testing. In X360 AI Tech, I came across an approach where the failure is looked at together with the execution context instead of treating the failed test as the problem straight away.

Things like logs, screenshots, traces, test steps, previous results, and application behaviour can help give a clearer picture of what actually happened.

I think this becomes even more important when AI is involved.

If AI sees a failed test and immediately changes something to make it pass, we might get a green result without really understanding what caused the failure in the first place.

For me, the useful question isn’t just: How do we fix this failed test?

It’s: What actually caused this test to fail?

Because sometimes the test isn’t the thing that needs fixing.

Can a Test Be Reliable and Still Be Outdated?

Sri Ramya — Mon, 21 Sep 2026 17:47:23 +0000

I’ve been thinking about something that’s easy to miss with test automation: can a test be completely reliable and still be outdated?

Tools like mabl focus on keeping coverage current as applications change, Testim uses AI-powered locators to make tests more resilient, Functionize looks at AI-driven maintenance and test realignment, and Applitools takes a different angle with visual and functional validation. I like that each approach is solving a different part of the same maintenance problem.

While exploring X360 AI Tech, this made me look at test maintenance a little differently. A test can keep passing every day and still be less useful if the requirement, user flow, or expected behavior has changed. In that case, there’s no failure telling us that something needs attention.

That’s what I find interesting about AI-assisted testing. Maybe AI shouldn’t only help us fix tests when they break. It could also help us notice tests that have been passing for a long time but may no longer represent what we actually want to validate.

I’m starting to think that a healthy test suite isn’t just one with fewer failures. It’s one where we still understand why each important test exists and whether it matches the current product.

How do you usually spot tests that are still passing but have slowly become outdated?