DEV Community: Lavanya

AI Agents Don’t Hesitate And That’s a Security Problem

Lavanya — Tue, 24 Mar 2026 08:29:02 +0000

AI Agents don’t hesitate.
They don’t second-guess.
They dont pause.

We didn’t just add AI to our stack. We gave it access, autonomy, speed. Which means mistakes are no longer small or slow.

Earlier this year, an internal AI coding agent at AWS ended up deleting and recreating parts of a production environment, causing a 13-hour outage!

An agent with too much access and zero hesitation.
This wasn’t an edge case, it’s a pattern.

Now compare that to something less accidental.

A red-team exercise showed how an autonomous agent could break into McKinsey’s internal AI platform, Lilli.

No credentials. No insider access.

Within hours, it was able to:

map internal APIs
identify a classic SQL injection and
escalate access across the system

AWS was accidental. But this is proof of risk.

In the AI era, the threat landscape is changing and rapdily; AI agents autonomously selecting and attacking targets will become the new normal.

So the question is “how fast?”

Vulnerabilities like SQL injection aren’t new. What’s changed is how easily they can be exploited now.

An agent can:

smartly identify

test

refine payloads and

chain access, all without stopping.

That is the real shift.

We’re not just writing vulnerable code anymore. We’re deploying systems that can:

find vulnerabilities, exploit them and amplify the impact automatically!

End-to-end. Without human friction.

When a developer makes a mistake, it’s usually contained and it takes time to escalate and someone eventually notices.

But when an AI agent makes a mistake, it executes end-to-end across systems at machine speed before you know it.

Most teams are focused on how to integrate agents and which model to use.

Are we thinking “What can this agent actually do if it goes wrong?”.

Almost all agents are over-permissioned.

In practice, agents are rarely given isolated, scoped identities.

Instead, they run on shared API keys or service accounts, like a CI/CD token with access to your repo, pipelines, and cloud which gives them far more access than they actually need.

Modern AI systems operate across multiple systems in a single flow, increase coupling between data, prompts, and execution paths.

So a “simple” bug can give access to the application and influence how AI behaves, making it much easier to exploit at scale.

Over the past few weeks, multiple incidents have shown the same pattern:

agents executing actions they shouldn’t

systems trusting inputs they shouldn’t

access levels that were never properly scoped

Do a Dev Sanity Check (Bookmark This)

If you’re working with agents (LangChain, AutoGPT, internal tools), run through this:

1. Identity Scope
Does each agent have its own identity? Or are you using shared service/admin tokens?
If one agent is compromised, what else breaks?

2. Destructive Actions
Can an agent execute DELETE / DROP / RECREATE?
If yes, is there a human approval gate?

3. Shadow Agents
Are devs running local AI tools with access to:
your repo, internal APIs and tokens. You probably have more agents than you think.

4. Blast Radius Mapping
Ask one question:
“If this agent goes rogue, what’s the worst it can touch?”

If the answer is one of “everything”, “production”, “our data layer”, you don’t have an AI problem. You have an access design problem.

You found a vulnerability. Cool.
But, can an agent exploit and chain this automatically?

Because that’s the real bar now.

This is the shift we’re seeing across the industry:
moving from finding vulnerabilities to proving real-world exploitability.

Gartner notes that offensive security trends are moving toward Preemptive security. Preemptive cybersecurity solutions (example Siemba) defend more effectively against such sophisticated AI-enabled exploits.

This proactive approach represents a shift from traditional reactive methods to preemptive measures aimed at mitigating risks before they escalate.

Automating a Web Application using Testsigma's NLP Grammar

Lavanya — Tue, 28 May 2019 18:32:20 +0000

The article explains how a sample web application like the OrangeHRM(an HR and Leave Management System) can be automated using the NLP based Test Automation tool, Testsigma.

We will do the following:

Navigate to the OrangeHRM website
Enter the Username(Admin) the Username field
Enter the Password(admin123) in the Password field
Click on Login

When you create a trial account with Testsigma, you will have a default Project named, Testsigma with a few samples. You may create a new Test Case in the existing Project or create a new Project and add test cases to this.

Now click on Add Test Steps to start writing the Test Steps for each of the 4 steps that we need to perform.

Step 1: Navigate to the application

The first step is quite easy. Just type/copy and paste the URL of the application you wish to automate in the Go To field and hit Start!

The first Test Step will automatically be added.

Step 2: Enter the Username

Now, to test the login scenario of this application. We will start by entering the Username.

Start typing, "Enter" and a list of grammar suggestions will appear.
From the list, choose the NLP grammar, “Enter test data in the ui identifier field” since this closely relates to what we need to perform.

We see 2 keywords in the grammar; a test data and a ui identifier. 'Test data' are inputs we provide like your Name, or the Username in this example, etc.

A 'ui identifier' is the path we use to “identify” an element(the Username field) anywhere on the website or the web page so as to perform any action on that element on the web page.

Now, you only need to replace the test data part with the value, “Admin”. Also, replace the ui identifier part with some meaningful variable like ‘UsernameLogin’. We will assign the unique ui identifier/path of the Username field to this variable.

You may as well skip this part and continue to add the Test Steps and later edit the ui identifier values. This is particularly useful if you are performing Test Automation in-sprint.

Let's assign the ui identifier its value right away. If you are familiar with creating xpaths, you may manually create and store the ui identifier value or you could use Testsigma’s Chrome extension that will automatically identify and store the exact path of the Username field for you. Here’s how you do this using the Chrome extension.

Now, click 'Create and Continue' to continue adding next Test Steps.

Step 3: Enter the Password

Now, enter the Password in the Password field. Use the same NLP grammar as the Username, “Enter test data in the ui identifier field”.
Replace test data with admin123 and replace ui identifier with some variable say, ‘PasswordforLogin’. i.e, “Enter admin123 in the PasswordforLogin field”.

Like before, you may either use the Chrome extension to get the ui identifier value of the ui identifier, ‘PasswordforLogin’ or you could try entering the ui identifier value manually by inspecting the Password field element.

Click on Create and Continue.

Step 4: Perform a click operation

Now, let's see how we perform the click operation to perform Login.
There’s no need to enter a test data value in this case, right? It’s just clicking on the Login button.
So, the NLP grammar is just “Click on ui identifier”. Replace the ui identifier with a name say, "Login" and assign its ui identifier value like we did previously.

Click on Create.

Step 5: Verify something

Here’s a bonus step! Let us also verify if some text is displayed in the page or not.
Click on Add Next Step. You just need to use the grammar, “Verify that the current page URL is test data”. You may try other NLP grammar suggestions too. Replace test data with your expected value.

To know what grammar to use, just refer the NLP grammar link at the left pane of the tool or refer the Help section(a ? symbol right next to the Test Steps) for the examples related to an action to be performed like Enter or Click.

Executing the Test Steps

Now, let’s run these 5 Test Steps. Just click on Run at the top right of the Test Steps page for a real quick execution of this test case and click Create and Run.

Once the test is run, check the Run Results. You will see the details of the execution with detailed test logs, screenshots and videos.

If you’d like to learn more about this in detail, create a free trial account with Testsigma for 30 days and see if it works for you.

Smart Test Automation using NLP

Lavanya — Wed, 22 May 2019 14:55:46 +0000

Development is fast-paced, so should be Testing to keep up with the accelerated delivery cycles. But Testing has not evolved at the same pace as Development. Even today, testers use age-old testing methods and tools to test complex applications.

Testing teams need to adopt automated test processes to keep up with the DevOps/CD advancements. As compared to Manual Testing, Automation Testing is best suited for changing application needs, to improve test effectiveness and coverage.

However, a few are still resistant towards Automation Testing mainly due to the huge scripting involved. Writing automated tests and maintaining the test scripts isn’t easy. This could be a bit of a struggle for someone new to Test Automation.

Automation Testers write super complex test scripts for each test case and are required to have a deep technical understanding of the underlying framework, application functionalities, programming knowledge, etc. It may take several lines of code to test even a small feature of an application.

Depending on the application type, Automation Testing requires expertise in multiple languages and technologies and a sudden shift towards Automation Testing can be a bit overwhelming for functional testers.

Also, Automation requires that the technical experts continuously monitor and maintain automated tests. Eventually, test automation becomes complex and unaffordable with an increased ROI time, especially, for small businesses.

A better approach with better collaboration and easy test authoring would be Scriptless Test Automation.

Scriptless Test Automation Approach

The biggest challenges in Automation Testing like we discussed is the initial time spent on test script development. The scriptless approach can speed up testing with better participation also, maximize the reusability of code with reduced maintenance.

Scriptless Automation tools hide/abstract the underlying code making the test automation process real easy and effortless. This will be particularly beneficial to Manual testing teams to automate complex applications efficiently even without scripting skills. With a huge part of the reusable tasks automated, they can also do a few rounds of Exploratory Testing.

This approach towards automation reduces time and effort required to test and ensure higher levels of maintainability and ease of updating changing requirements to fit into the Agile/DevOps process.

Benefits of going Scriptless

A Unified approach to diverse application types on multiple browsers and platforms
Less or no learning curve
Easy updating of changes and updates
Test scripts can be reviewed by domain experts and business analysts
A reduced test authoring time and faster Return On Investment

The main challenges with legacy test automation are building a robust and maintainable automation framework and managing the objects and identifiers.

Automated tests need to be reusable and maintainable and should not be resistant to changes in the application UI and should ensure that the automated tests do not break. Today with "self healing" tests that automatically takes care of flaky tests, this is no longer an issue.

NLP in Testing

A new advancement in Scriptless Test Automation is Natural Language Processing(NLP) that is used to write simple tests in natural understandable language.

The application of NLP reduces test creation time and is done in a format that is intuitively easy to for anyone to quickly adapt and use. Tools like Testsigma use simple plain English to automate tests and the changing UI elements are maintained by AI.

The tests are as simple as "Go to https://testsigma.com”,
“Enter Name in the Username field”,
“Verify that the page displays text Testsigma, etc.

NLP techniques were initially considered puerile and inadequate. However, over the years, NLP has advanced. Some of our day to day applications like Siri, Alexa, autocomplete in emails, etc that try and cut down our manual efforts are a few examples of this.

The NLP based tests can accommodate changes anytime and can easily be edited(if required). NLP makes it easier to automate all of your test cases and tightly integrate them with your dynamic delivery pipeline while optimizing user experience.

Let’s look at a few additional features that would boost test automation.

1. Smart Element locators

Once the Tests are created and ready, Test maintenance is the next hurdle a tester has to go through. Undesirable Test maintenance wastes productive time that could be used for manual feature checks and increasing test coverage.

It ensures that the Functional Tests doesn’t break even if the product undergoes radical change. The tests will stay strong in the tide of code changes.

2. Conditional waiting

Conditional waits allow avoid ‘dead waits’ that pauses the Execution for a specified amount of time even if the condition we are waiting for becomes true halfway.

NLP based test automation tool, Testsigma has an extensive set of Conditional Wait statements which helps to add waits based on user-defined conditions:

Wait until the current page is loaded completely
Wait until the element ui identifier is clickable
Wait until the element ui identifier is selected
Wait until the element ui identifier is visible & many more…

3. Conditional flow and Control Structures

Controls Structures provided by programming languages is one of the most powerful features available in Selenium. This allows defining different test flows to loop a set of Test Steps multiple times based on specific conditions.

Imagine the same set of features available in a Scriptless tool without the added complexity.
With Testsigma, you can either loop on a whole Test Case with a set of parameters(data-driven approach) or loop on a subset of Test Steps. This is especially useful if we want to simply repeat a step multiple times with/without an available Test Data set.

4. Easy Assertions in scriptless test automation

Creating the navigation steps alone doesn’t add any value to the Tests. The real value comes with the Assertions. Assertion verifies that the required business logic is implemented properly and works in a way the Product owner wants it to.

Adding Assertions in Testsigma is similar to adding Conditional waits – using already available Verification statements.

Here’s a set of Assertions available by default:

Verify that an Alert with text test data is displayed
Verify that the ui identifier checkbox is disabled
Verify that the ui identifier list has option with text test data selected
Verify that the Alert displays the message test data
Verify that the count of elements identified by locator ui identifier is test data and lot more.

5. Modifications without redo

Record-and-Playback tools do not permit easy editing of recorded tests. The Tester has no choice but to re-record the Test Steps from the beginning of the flow if the application changes even slightly. This is far from ideal. This is explained more in detail in Why is Record and Playback not suitable for enterprise-class Test Automation?

Like we mentioned earlier, Modification is a piece of cake when it comes to NLP based test scripts and permits to add Test Steps at the beginning, end or anywhere in between easily without redo.

6. Reusable Steps in Scriptless Test Automation

In scenarios similar to "Login" of an an application, it would be a redundant effort to create the same steps for each of the Test Cases. Therefore, it is necessary to provide some mechanism with which we can group a set of Test Steps for repeated use.

7. Cross-browser Support

With a plethora of devices(Desktop or Mobile) used by end users, we need to make sure the app functions as intended on all those devices. You may use Analytics tools such as Google Analytics to find out device usage patterns.

Now with Cloud Based Scriptless Testing solutions, automated tests can be run on thousands of real Device/OS/Browser versions combo and also local physical devices.

8. Test Reporting and Analytics

Testsigma provides a detailed Test Report for all the Test Case executions that consists of a powerful visual feedback system with step by step visual reports.

Each of the Test Step has a passed/failed status indicator and screenshots that denote the state of the Application when the failure occurred.

In some cases, the Test Case can fail at a step and the error might propagate down unnoticed until the next assertion. Step by step analysis in Scriptless Test Automation tools like Testsigma includes Screenshots and logs that helps us to understand the actual step where the Test failed.

You can easily figure out what went wrong using the Test Execution video logs that help pinpoint the step which deviated from the actual Test Case also gives a quick analysis as to how our tests are performing over a period of time.

9. Ability to insert code

If you need to use the power and flexibility of Code, you can create Custom Functions to write code and create your custom test scripts in addition to the NLP based grammar steps.

10. Continuous Integration

With necessary third party tools' integrations(Team Collaboration tools, Bug Tracking tools, ALM tools and CI tools), you could bring all your testing resources together thereby achieving Continuous Testing.

In addition to plain natural language test creation, a few additional features that Testsigma offers include:

Possible Failure Detection – Affected Test Cases
Test Management Capabilities(Requirements, Test Suites, Test Case priorities e.t.c)
Automatic Bug Reporting(JIRA)
Shared Object Repository
State Management

With the simplicity of NLP based tests, flexible, easily understandable tests can be created which well suits Agile development and allows for iterative delivery of quality software!

Image credit: Business vector created by freepik - www.freepik.com