DEV Community: Let's Automate 🛡️

AI-Assisted Testing vs AI Agents vs AI Agent Skills: A Practical Journey Through All Three

Let's Automate 🛡️ — Sat, 07 Mar 2026 13:08:54 +0000

Most teams are only using one layer of AI in testing. Here is what the full picture looks like — and how I built across all three.

Photo by Possessed Photography on Unsplash

Before any of this made sense, I had to answer a more basic question: what does AI QA Engineering actually mean?

What is AI QA Engineering — and Why QAEs, SDETs, and QA Automation Engineers Should Pay Attention

And before touching AI at all — the foundations still matter. Clean BDD tests. Reports that stakeholders can read.

How to Add Beautiful BDD Test Reports to Your Reqnroll Project Using Expressium LivingDoc

Before you automate smarter, you have to know what good looks like.

Layer 1 — AI-Assisted Testing

AI speeds you up. You are still driving.

This is where most teams start — and where most teams stay.

You write a prompt, get a test, review it, ship it. AI is a productivity multiplier. GitHub Copilot suggests the next line. ChatGPT drafts your test cases. Claude rewrites a flaky selector. You are in control at every step.

The catch? A bad prompt gives you a bad test — and it will look convincing. Garbage in, confident garbage out.

Crafting Effective Prompts for GenAI in Software Testing

I built ai-natural-language-tests at this layer. Give it a plain English requirement, and it generates Cypress or Playwright tests using GPT-4, LangChain, and LangGraph. Every output still needs your eyes on it — but the heavy lifting is done.

Same idea with JIRA-QA-Automation-with-AI : feed it a JIRA story with acceptance criteria, and BDD test scripts come out the other side. Human judgment still required at the end. You own every decision.

That last part is the definition of this layer.

Layer 2 — AI Agents for Testing

You give the goal. The agent executes, adapts, and decides.

At this layer, you stop steering and start delegating.

You set the objective. The agent figures out how to get there — and when something breaks mid-run, it handles that too. No human in the loop for every step.

selenium-selfhealing-mcp is a good example of what this looks like in practice. A UI change breaks a Selenium locator mid-execution. The agent inspects the DOM, finds the updated element, and keeps going — without stopping to ask you what to do. I submitted this to the Docker MCP Registry, and watching it recover from failures on its own still feels like a step-change from Layer 1.

For .NET teams, SeleniumSelfHealing.Reqnroll does the same with C#, NUnit, Reqnroll, and Semantic Kernel. And IntelliTest takes it further — write your assertions in plain English, and the agent decides whether the application behaviour actually matches the intent.

But there is a trap at this layer. Agents move fast and look thorough. It is easy to trust the output and skip the checks. Coverage looks complete — but the agent may have tested the wrong thing entirely.

The AI QA Engineer’s Decision Framework: When NOT to Use AI in Testing

And if you are using AI agents to run tests, a harder question follows: how do you know the agent’s output is correct? That is the LLM evaluation problem, and it turns out to be one of the most interesting unsolved problems in this space.

LLM Evaluation Explained: How to Know If Your AI Is Actually Working

Layer 3 — AI Agent Skills

Not a tool. Not an agent. Expertise that travels.

Layer 3 is the one most people have not thought about yet.

Here is the pattern I kept running into: every new agent project started from scratch. New codebase, new prompts, same underlying knowledge — how to read a requirement, what makes a test meaningful, when to flag a risk. The expertise was always being rebuilt. That seemed wrong.

A skill is a portable, encoded unit of expertise. It is not tied to one agent or one project. Any compatible agent can load it and apply it — without rebuilding the logic again. You build it once, and it travels.

GitHub Copilot Agent Skills: Teaching AI Your Repository Patterns

vibe-coding-checklist applies the same idea to AI code review — a shared quality framework that any team or any agent can use consistently.

The shift in thinking is subtle but significant. At Layer 1, you build prompts and tools. At Layer 2, you build goals and trust boundaries. At Layer 3, you build expertise itself — in a form that outlasts any single project or team.

The Difference That Matters

AI-Assisted Testing vs AI Agents vs AI Agent Skills

Three layers. All called AI testing. Now you know which one you are actually in.

All repos → github.com/aiqualitylab

More writing → aiqualityengineer.com

The GitHub Copilot Features That Are Quietly Draining Your Premium Requests

Let's Automate 🛡️ — Thu, 19 Feb 2026 17:19:23 +0000

10 optimisations most developers miss — including why the Copilot Coding Agent beats Agent Mode Chat every time

Most developers hit their monthly limit in the first week. Here’s what’s actually happening under the hood — and how to work smarter before it happens to you.

Photo by Resume Genius on Unsplash

Before diving in, it helps to understand what GitHub Copilot actually counts as a premium request, because most developers don’t find out until it’s too late.

Inline code completions on paid plans are unlimited and cost nothing. What drains your monthly allowance is everything else — Copilot Chat, Agent Mode, Copilot Code Review, Copilot CLI, and the Copilot Coding Agent.

Each model also carries a multiplier. Some models are included free on paid plans. Once your allowance is gone, premium features are locked for the rest of the billing cycle.

Knowing that, here’s how to make every request count.

1. Name your functions like they’re instructions

Inline autocomplete is unlimited on paid plans and costs nothing from your premium allowance. The more precisely you name a function, the more accurately Copilot completes the body without any Chat involved. This is your primary tool, not a fallback.

2. Write your intent as a comment above the cursor

A detailed comment placed directly before your cursor is treated by Copilot as an instruction. You get the same outcome as a Chat message at zero premium cost. Use this for any logic you would otherwise describe to Copilot in conversation.

3. Cycle through alternatives with Alt+] before opening Chat

When the first inline suggestion misses, most developers immediately reach for Chat. Before doing that, cycle through alternative suggestions. The second or third option is often exactly what’s needed — and one saved Chat message multiplies across a full day of work.

4. Disable Agent Mode when you’re not actively using it

Agent Mode runs in the background and silently runs even when you’re not directing it. GitHub’s official documentation explicitly flags this as a common cause of unexpected quota drain. Disable it in your repository settings when it isn’t part of your current workflow.

5. Use the Copilot Coding Agent for complex tasks instead of Agent Mode Chat

This is one of the least-known optimisations available. The Copilot Coding Agent — the one that creates and modifies pull requests asynchronously — counts as one premium request per full session regardless of how much work it does. Agent Mode Chat charges one premium request per message, multiplied by the model rate. For any task involving multiple files or significant implementation work, the Coding Agent is dramatically more efficient.

6. Start a new Chat thread when switching topics

As a conversation grows, all prior messages remain in context and contribute to token consumption. GitHub’s documentation specifically calls this out as a driver of elevated usage. When you move to a new task or a different area of your codebase, start a fresh thread rather than continuing an existing one.

7. Understand the model multiplier before choosing one

Before switching to a powerful model, weigh whether the capability gain justifies the cost. For most day-to-day work, it doesn’t.

8. Use auto model selection for a built-in discount

When you enable auto model selection in Copilot Chat in VS Code, GitHub applies a 10% multiplier discount across all premium model usage. It requires no change to your workflow and the saving compounds quietly across a full month.

9. Use #file references instead of @workspace

@workspace scans your entire codebase on every message, consuming more than most questions require. Using #file:yourfile.ts targets exactly the context Copilot needs, which produces more focused answers with less back-and-forth and fewer requests spent getting there.

10. Set a budget alert before your allowance runs out

GitHub lets you configure alerts at 75%, 90%, and 100% of any spending threshold you define. Setting a low or zero spending budget with alerts enabled means you get notified well before premium features are cut off — without risking unexpected charges. Check your current usage anytime at github.com/settings/billing or through the Copilot icon in your IDE status bar.

The Principle Underneath All of It

Every tip here points back to the same question worth asking before you open Chat: is there a way to get this through autocomplete instead?

Reference — https://docs.github.com/en/copilot

Most of the time, there is. And building that habit is what separates developers who hit the wall in week one from those who reach month end with room to spare.

AI Natural Language Tests — Dual Framework Test Automation with Cypress & Playwright

Let's Automate 🛡️ — Sun, 01 Feb 2026 16:55:23 +0000

AI Natural Language Tests — Dual Framework Test Automation with Cypress & Playwright

Open-source AI test automation framework with natural language test generation, self-healing, and dual framework support

Writing end-to-end tests is one of those things every team knows they should do, but nobody really enjoys doing. You stare at a login page, figure out the selectors, write the steps, handle the waits, and repeat this for every feature. I kept thinking — what if I could just say what I want to test, and let AI handle the rest?

That’s exactly what I built.

Architecture

What Is It?

ai-natural-language-tests is an open-source tool that takes a plain English description of a test scenario and generates a fully working Cypress or Playwright test file. No templates. No copy-pasting. You describe the test, point it at a URL, and it writes the code.

Here’s what a typical command looks like:

python qa_automation.py "Test login with valid credentials" --url https://the-internet.herokuapp.com/login

That single line does everything — fetches the page, reads the HTML, picks up the right selectors, and generates a complete test file you can run immediately.

Want Playwright instead of Cypress? Just add a flag:

python qa_automation.py "Test login with valid credentials" --url https://the-internet.herokuapp.com/login --framework playwright

How It Actually Works

Under the hood, the tool runs a 5-step workflow built with LangGraph:

Complete Workflow

Step 1 — It sets up a vector store. Think of this as a memory bank for test patterns.

Step 2 — It fetches the target URL, pulls the HTML, and extracts useful selectors like input fields, buttons, and links.

Step 3 — It searches the vector store for similar tests it has generated before. If you tested a login page last week, it remembers the patterns.

Step 4 — It sends everything to GPT-4 along with a carefully crafted prompt — the description, the selectors, and any matching patterns from history. The AI generates the actual test code.

Step 5 — Optionally, it runs the test right away using Cypress or Playwright.

The interesting part is Step 3. Every test the tool generates gets saved as a pattern. Over time, it builds a library of patterns and uses them to write better tests. The first test for a login page might be decent. The tenth one will be much better because it has learned from all the previous ones.

Why Two Frameworks?

I started with Cypress because it’s what most teams I’ve worked with use. But Playwright has been gaining serious traction — especially for teams that need multi-browser testing or prefer TypeScript.

So in v3.1, I added full Playwright support. The tool uses different prompts for each framework. The Cypress prompt focuses on chaining commands and cy.get() patterns. The Playwright prompt covers locators, async/await, network interception, multi-tab handling, and all the TypeScript-specific patterns.

You pick the framework. The AI adapts.

The Part I Didn’t Expect — Failure Analysis

While building this, I realized that generating tests is only half the problem. Tests fail. And reading Cypress or Playwright error logs can be painful, especially for someone newer to the frameworks.

So I added an AI-powered failure analyzer:

python qa_automation.py --analyze "CypressError: Timed out retrying after 4000ms"

It reads the error, explains what went wrong in plain language, and suggests a fix. You can also point it at a log file. It’s a small feature but it has saved me a surprising amount of time.

Running It in CI/CD

The tool comes with a GitHub Actions workflow out of the box. You can trigger it manually from the Actions tab — type your test description, provide a URL, pick Cypress or Playwright, and it runs the full pipeline. Generate, execute, and get results — all inside your CI.

CI/CD PIPELINE

This makes it practical for teams that want to try AI-generated tests without changing their existing setup. Just add the workflow and trigger it when you need a new test.

What I Learned Building This

A few things surprised me along the way:

Prompts matter more than the model. I spent more time refining the system prompts than on any other part of the codebase. A well-structured prompt with clear constraints produces dramatically better test code than a vague one, regardless of which GPT model you use.

Pattern learning is underrated. The vector store approach turned out to be more useful than I expected. When the tool has seen similar pages before, the generated tests are noticeably more accurate. It picks up things like common selector patterns and assertion styles from its history.

Keeping frameworks separate is important. Early on, I tried using a single generic prompt for both Cypress and Playwright. The results were mediocre for both. Dedicated prompts for each framework made a huge difference in output quality.

Try It Out

The project is open source and ready to use:

GitHub: github.com/aiqualitylab/ai-natural-language-tests

First Release — https://github.com/aiqualitylab/ai-natural-language-tests/releases/tag/v2026.02.01

Setup takes about five minutes — clone the repo, install dependencies, add your OpenAI API key, and you’re generating tests.

If you work in QA or test automation and you’ve been curious about how AI fits into your workflow, give it a try. I’d love to hear what you think.

Exploring how AI can make quality engineering more practical and less tedious. I write about this stuff regularly at AI Quality Engineer .

The AI QA Engineer’s Decision Framework: When NOT to Use AI in Testing

Let's Automate 🛡️ — Sun, 25 Jan 2026 10:47:51 +0000

A Practical Guide for Quality Engineers Who Want Results, Not Hype

When NOT to Use AI in Testing: A Simple Guide

Stop. Think. Then Decide.

The Big Question

Everyone talks about using AI in testing. But nobody talks about when to SKIP it.

This guide helps you decide: AI or no AI?

Why This Matters

AI testing sounds cool. But it comes with baggage:

It costs money — AI tools need servers, licenses, and API calls.

It needs babysitting — Models drift. Prompts need tuning. Things break in weird ways.

It’s hard to debug — When AI tests fail, figuring out WHY is painful.

Your team might forget basics — If AI does everything, manual debugging skills fade.

AI isn’t bad. But it’s not always the answer.

7 Times to Skip AI (Use Traditional Testing Instead)

1. Math and Calculations

Example: Tax calculators, loan interest, pricing formulas.

Why skip AI? The answer is either right or wrong. No guessing needed. No patterns to learn.

Do this instead: Simple data-driven tests. Input goes in. Expected output comes out. Done.

2. Audit and Compliance Systems

Example: Banking apps, healthcare records, legal documents.

Why skip AI? Auditors want proof. They want to see EXACTLY what you tested. AI is unpredictable — same prompt, different results.

Do this instead: Scripted tests with detailed logs. Every step recorded. Every result traceable.

3. Speed and Load Testing

Example: Can your app handle 10,000 users at once?

Why skip AI? You’re measuring app speed. AI adds its own delay. You’d be measuring AI, not your app.

Do this instead: Use tools built for this — JMeter, k6, Gatling. They’re fast and focused.

4. Basic CRUD Operations

Example: Create user. Read user. Update user. Delete user.

Why skip AI? It’s simple. AI is overkill. Like using a rocket to go to the grocery store.

Do this instead: Write one test template. Copy it for each operation. Fast and easy.

5. Screens That Never Change

Example: Internal admin panels. Old systems nobody touches.

Why skip AI? AI shines when things CHANGE. Self-healing locators fix moving targets. No movement? No need.

Do this instead: Regular automation. Page Object Model. Set it and forget it.

6. Security Testing

Example: Finding SQL injection, XSS attacks, login bypasses.

Why skip AI? Security needs creative thinking. Breaking things in new ways. AI follows patterns — hackers don’t.

Do this instead: Security tools (OWASP ZAP, Burp Suite) plus human testers who think like attackers.

7. Physical Device Testing

Example: Barcode scanners, payment terminals, IoT sensors.

Why skip AI? AI lives in software. It can’t press physical buttons or read blinking lights.

Do this instead: Hardware test rigs. Human testers. Real-world verification.

The Quick Decision Guide

Ask yourself these 4 questions:

DECISION TABLE FRAMEWORK

Before You Buy Any AI Tool, Answer These:

What exact problem am I solving? (Not “we want AI” — a real problem)

Can a simple script fix this? (Seriously, can it?)

How will I know if it worked? (What number goes up or down?)

Who will maintain it? (AI tools need constant care)

Can I explain it to my boss? (If you can’t explain it, don’t buy it)

The Simple Truth

AI is a tool. Not a magic wand.

Good testers know WHEN to use each tool:

USAGE CHECKLIST

One Page Summary

USE AI FOR:

Generating test ideas from requirements

Handling UI changes automatically

Analyzing why tests keep failing

Creating test data variations

Exploring edge cases

SKIP AI FOR:

Exact calculations (math, money, dates)

Compliance and audit trails

Performance/load measurements

Simple CRUD operations

Stable, unchanging systems

Security penetration testing

Physical hardware testing

Final Word

The smartest move isn’t always the newest tool.

Sometimes a simple script beats a fancy AI.

Know when to use AI. Know when to skip it. That’s real skill.

Machine Learning Pipelines Made Easy for Quality Assurance Professionals

Let's Automate 🛡️ — Sat, 10 Jan 2026 19:45:18 +0000

A very simple guide to how machine learning works

Machine learning looks hard. But it is not.

If you know QA, you already know the basics.

ML systems have three parts. We call them FTI:

F = Feature (clean the data)

T = Training (teach the model)

I = Inference (use the model)

Let me explain each one.

Part 1: Feature Pipeline

What does it do?

It cleans dirty data.

Simple example:

You have messy data. Names are written in different ways. Dates are in wrong formats. Numbers have errors.

This pipeline fixes all that. It makes data clean and ready.

Feature Pipeline Detail

In QA words:

You never test with bad data. You clean it first. This pipeline does the same thing.

The clean data goes to a Feature Store.

Part 2: Training Pipeline

What does it do?

It teaches the model.

Simple example:

You show the model 1000 pictures of cats. You tell it “this is a cat” each time. The model learns what a cat looks like.

In QA words:

You learn from requirements. Then you write test cases. The model learns from data. Then it can make predictions.

Picture:

The smart model goes to a Model Registry.

Training Pipeline Detail

Part 3: Inference Pipeline

What does it do?

It uses the model to answer questions.

Simple example:

Someone shows a new picture. The model says “this is a cat” or “this is not a cat.”

In QA words:

This is like running tests in production. The model is working and giving answers.

Inference Pipeline Detail

Two Important Storage Places

Feature Store

Keeps clean data

Saves old versions

Everyone uses same data

Model Registry

Keeps trained models

Saves old versions

You know which model is in production

The Full Picture

Full FTI Pipeline Overview

Why This is Easy for QA

You already know:

✓ How to check data quality → Test Feature Pipeline

✓ How to compare old vs new → Test Training Pipeline

✓ How to test in production → Test Inference Pipeline

Five Things to Remember

Three parts. Feature, Training, Inference. That’s it.

Clean data is key. Bad data = bad model.

Save everything. Keep old data. Keep old models. You can go back if needed.

Test each part. Don’t test everything together. Test one part at a time.

Your skills work here. QA testing skills work for ML testing too.

Last Words

ML is just software with a learning step.

You already know how to test software. Now you can test ML too.

Start simple. Ask: “Show me the three pipelines.”

Then test each one.

You can do this.

I Built an AI-Powered Test Data Generator That Analyzes Any URL and Creates Test Data JSON

Let's Automate 🛡️ — Wed, 31 Dec 2025 19:12:47 +0000

I got tired of manually inspecting HTML to find selectors. So I taught my framework to do it instead.

Architecture flow

Here’s a question that kept me up at night:

Why am I spending more time finding selectors than writing actual tests?

I watched myself burn 30 minutes on a simple login test — not writing the test itself, but hunting through DevTools for the right selectors, creating fixture files, and crafting test data that would actually work.

What if the framework could just… look at the page and figure it out?

The Problem Nobody Talks About

Here’s the dirty secret of test automation: writing the actual test is the easy part.

The hard part? Finding #username vs input[name="user"] vs .login-field. Creating realistic test data. Building fixture files that match the actual form structure.

Every new page means:

Open DevTools

Inspect elements

Copy selectors

Hope they’re stable

Create JSON fixtures

Hope nothing changes tomorrow

Most “AI-powered” testing tools focus on running tests or analyzing failures. But what about the beginning — the tedious setup that drains your time before you write a single assertion?

The Experiment: Teaching AI to See

The idea was simple but audacious: give the AI a URL and let it figure out everything else.

Not mock data. Not hardcoded selectors. Real selectors from real HTML.

Here’s what I wanted:

python qa_automation.py "Test login" --url https://the-internet.herokuapp.com/login

And the framework should:

Fetch the actual page

Analyze the HTML structure

Extract real, working selectors

Generate meaningful test cases

Save everything as a Cypress fixture

Then generate tests that use that data

Sounds impossible? I thought so too.

How It Actually Works

The magic happens in about 50 lines of Python:

def generate_test_data_from_url(url: str, requirements: list) -> tuple:
    # Step 1: Fetch the real page
    resp = requests.get(url, timeout=10, headers={'User-Agent': 'Mozilla/5.0'})
    html = resp.text[:5000] # First 5KB is usually enough

    # Step 2: Ask AI to analyze it
    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

    prompt = f"""Analyze this HTML and generate test data.

    URL: {url}
    HTML: {html}

    Return JSON with:
    - Real selectors from the HTML
    - Valid test case with working data
    - Invalid test case for error handling
    """

    # Step 3: Parse and save as fixture
    test_data = json.loads(llm.invoke(prompt).content)

    with open("cypress/fixtures/url_test_data.json", 'w') as f:
        json.dump(test_data, f, indent=2)

    return test_data

The AI doesn’t guess. It reads the actual HTML and extracts what’s really there.

Complete Workflow

What The AI Sees vs What It Returns

When I point it at a login page, here’s the actual flow:

Input: Just a URL

--url https://the-internet.herokuapp.com/login

What the AI analyzes:

<input type="text" id="username" name="username">
<input type="password" id="password" name="password">
<button type="submit" class="radius">Login</button>

What it generates:

{
  "url": "https://the-internet.herokuapp.com/login",
  "selectors": {
    "username": "#username",
    "password": "#password",
    "submit": "button[type='submit']"
  },
  "test_cases": [
    {
      "name": "valid_test",
      "username": "tomsmith",
      "password": "SuperSecretPassword!",
      "expected": "success"
    },
    {
      "name": "invalid_test", 
      "username": "wronguser",
      "password": "badpassword",
      "expected": "error"
    }
  ]
}

Real selectors. Actual test data. Zero manual inspection.

The Generated Test Uses It All

The framework then generates a Cypress test that consumes this fixture:

describe('Login Tests', function () {
    beforeEach(function () {
        cy.fixture('url_test_data').then((data) => {
            this.testData = data;
        });
    });

it('should login with valid credentials', function () {
        cy.visit(this.testData.url);
        const valid = this.testData.test_cases.find(tc => tc.name === 'valid_test');

        cy.get(this.testData.selectors.username).type(valid.username);
        cy.get(this.testData.selectors.password).type(valid.password);
        cy.get(this.testData.selectors.submit).click();

        cy.url().should('include', '/secure');
    });
    it('should show error with invalid credentials', function () {
        cy.visit(this.testData.url);
        const invalid = this.testData.test_cases.find(tc => tc.name === 'invalid_test');

        cy.get(this.testData.selectors.username).type(invalid.username);
        cy.get(this.testData.selectors.password).type(invalid.password);
        cy.get(this.testData.selectors.submit).click();

        cy.get('#flash').should('contain', 'invalid');
    });
});

Notice something? The selectors come from the fixture, not hardcoded in the test.

If the page changes, update the fixture. Tests stay clean.

Two Ways to Feed Data

Sometimes you already have test data. Maybe from a previous run. Maybe from your team’s shared fixtures.

So I added a second option:

# Option 1: AI analyzes live URL
python qa_automation.py "Test login" --url https://example.com/login

# Option 2: Use existing JSON file
python qa_automation.py "Test login" --data cypress/fixtures/my_data.json

Same test generation. Different data sources. Your choice.

The Part That Surprised Me

I expected the AI to find basic selectors. What I didn’t expect was how well it understood context.

When analyzing a registration form, it didn’t just find #email — it generated test data like:

Valid: testuser@example.com

Invalid: not-an-email

For password fields:

Valid: SecurePass123!

Invalid: 123 (too short)

The AI understood what kind of data each field expected. Not because I told it — because it read the HTML attributes, labels, and validation patterns.

The Gotcha: Fixtures Need function() Syntax

One thing tripped me up for hours. Cypress fixtures with this.testData require a specific pattern:

// WRONG - arrow functions don't have 'this'
describe('Test', () => {
    beforeEach(() => {
        cy.fixture('data').then((d) => { this.testData = d; }); // undefined!
    });
});

// RIGHT - function() preserves 'this'
describe('Test', function () {
    beforeEach(function () {
        cy.fixture('data').then((data) => { this.testData = data; });
    });

    it('works', function () {
        console.log(this.testData); // actual data!
    });
});

The framework now enforces this pattern in generated tests. Lesson learned the hard way.

What This Means For Your Workflow

Before:

Open page in browser

Inspect elements manually

Copy selectors to notepad

Create fixture JSON by hand

Write test using those selectors

Fix typos in selectors

Run test

Debug why selectors don’t work

After:

Run one command with URL

Framework handles the rest

That’s not an exaggeration. The 30-minute login test? Under 2 minutes now.

Try It Yourself

The framework is open source:

git clone https://github.com/user/cypress-natural-language-tests
cd cypress-natural-language-tests
pip install -r requirements.txt

Set your API key:

export OPENAI_API_KEY=your_key_here
export OPENROUTER_API_KEY=your_openrouter_api_key_here

Generate tests from any URL:

python qa_automation.py "Test the login form" --url https://the-internet.herokuapp.com/login

Check what it created:

cat cypress/fixtures/url_test_data.json
cat cypress/e2e/generated/*.cy.js

The Bigger Picture

We’re at an interesting moment in test automation. The tooling is getting smarter, but the real breakthrough isn’t replacing testers — it’s eliminating the tedious parts.

Finding selectors is tedious. Creating fixture files is tedious. Debugging why #submit-btn worked yesterday but not today is tedious.

Let AI handle tedious. Let humans handle important.

That’s the framework I’m building.

Follow for more AI + QA experiments:

GitHub: https://github.com/aiqualitylab/cypress-natural-language-tests.git

I Built an AI-Powered Cypress Framework That Analyses Test Failures for Free

Let's Automate 🛡️ — Sun, 28 Dec 2025 14:03:59 +0000

Cypress test debugging is painful. This free AI-powered framework analyses failures instantly and tells you exactly what went wrong.

AI-Powered Cypress Framework That Analyses Test Failures for Free

Ever stared at a cryptic Cypress error message wondering what broke? 😩 We’ve all been there. That’s why I built something that changed my debugging workflow forever.

Introducing v2.1 of my Cypress Natural Language Test Framework — now featuring 🔍 AI Failure Analysis that costs you absolutely nothing.

😤 The Problem Every QA Engineer Knows

Picture this: Your CI pipeline fails and error be like:

CypressError: Timed out retrying after 4000ms: Expected to find element: '#submit-btn', but never found it.

Now you’re left guessing:

🤔 Did the selector change?

⏳ Is the page loading too slowly?

✏️ Did someone rename the button?

⚡ Is it a timing issue?

You spend the next hour digging through logs, comparing commits, and testing locally. Sound familiar?

💡 The Solution: AI That Debugs For You

With v2.1, debugging becomes a one-liner:

python qa_automation.py --analyze "CypressError: Timed out retrying: Expected to find element: #submit-btn"

Output:

🔍 Analyzing...
REASON: Element #submit-btn not found - selector likely changed during recent UI update
FIX: Use cy.get('[data-testid="submit"]') or add cy.wait() before the click action

✅ Two lines. Problem identified. Solution provided. Done.

🏗️ System Architecture

Here’s how the entire framework fits together:

⚙️ How It Works Under The Hood

The implementation is surprisingly simple. Here’s the core function:

def analyze_failure(log: str) -> str:
    response = requests.post(
        url="https://openrouter.ai/api/v1/chat/completions",
        headers={
            "Authorization": f"Bearer {os.getenv('OPENROUTER_API_KEY')}",
            "Content-Type": "application/json"
        },
        json={
            "model": "deepseek/deepseek-r1-0528:free",
            "messages": [{"role": "user", "content": f"Analyze this Cypress test failure. Reply ONLY:\nREASON: (one line)\nFIX: (one line)\n\n{log}"}],
            "max_tokens": 150
        }
    )
    return response.json()["choices"][0]["message"]["content"]

That’s it. About 15 lines of code that leverage OpenRouter’s free tier with DeepSeek R1. 🆓

🛠️ Three Ways To Use It

1️⃣ Direct from command line:

python qa_automation.py --analyze "Your error message here"

2️⃣ From a log file:

python qa_automation.py --analyze -f cypress-output.log

3️⃣ Piped from another command:

cat error.log | python qa_automation.py --analyze

🔄 CI/CD Integration

The real power comes when you integrate this into your pipeline. Here’s how the updated GitHub Actions workflow looks:

- name: Run Cypress tests
  id: tests
  continue-on-error: true
  run: |
    npx cypress run --spec "cypress/e2e/generated/**/*.cy.js" 2>&1 | tee test-output.log- name: AI Failure Analysis
  if: steps.tests.outcome == 'failure'
  env:
    OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
  run: |
    echo "Analyzing failures with AI..."
    python qa_automation.py --analyze -f test-output.log

When tests fail, your CI logs now include actionable insights instead of just error dumps. 📋

🚀 Setting It Up

Step 1: Get your free API key from openrouter.ai 🔑

Step 2: Add to your .env:

OPENROUTER_API_KEY=your_key_here

Step 3: Add requests to requirements.txt (if not already there) 📦

Step 4: Start analyzing 🎉

That’s the entire setup. No complex configurations. No paid subscriptions.

🖥️ Local Development Flow

For local development, the flow is just as smooth:

📦 What’s In v2.1

Here’s everything new in this release:

Feature Description

🔍 AI Failure Analyzer Instant debugging with free LLM

🌐 OpenRouter Integration Uses DeepSeek R1 at zero cost

💻 CLI Flag Simple --analyze command

📁 File Input Analyze entire log files with -f

⚙️ CI/CD Ready Updated GitHub Actions workflow

Combined with v2.0 features:

🤖 Natural language test generation

🔄 cy.prompt() self-healing tests

📊 LangGraph workflow orchestration

📚 Vector store documentation context

🌍 Real World Example

Old approach: Manual Investigation 😓

New approach:

python qa_automation.py --analyze -f nightly-run.log

Result:

REASON: Login button selector changed from #login-btn to .auth-button
FIX: Update selector to cy.get('.auth-button') or use data-testid

REASON: API response timeout - server took 6s, test timeout was 4s
FIX: Increase timeout with cy.request({timeout: 10000}) or add retry logic

REASON: Element detached from DOM after React re-render
FIX: Add cy.wait() after state change or use {force: true} option

🔗 Try It Yourself

The framework is open source and available now:

🔗 GitHub: github.com/aiqualitylab/cypress-natural-language-tests

Clone it, set up your API keys, and start generating tests and debugging failures with AI.

💭 Final Thoughts

AI shouldn’t just generate code. It should help maintain it too. This failure analyzer is my attempt at closing that loop — from requirements to tests to debugging, all AI-assisted.

The best part? It’s completely free to use. 🆓

Give it a try and let me know how much time it saves you! 💬

If this helped you, consider ⭐ starring the repo. It helps others discover it.

AI-Powered Cypress Test Generation from Natural Language v2.0 — Now with cy.prompt() Self-Healing

Let's Automate 🛡️ — Sat, 27 Dec 2025 11:46:37 +0000

AI-Powered Cypress Test Generation from Natural Language — Now with cy.prompt() Self-Healing

Transform plain English requirements into production-ready Cypress tests using GPT-4, LangChain, and LangGraph — run locally or in CI/CD

My Open-source project: github.com/aiqualitylab/cypress-natural-language-tests, which utilizes Cypress’s official AI-powered cy.prompt() command introduced at CypressConf 2025.

AI-Powered Cypress Test Generation from Natural Language v2.0 — Now with cy.prompt() Self-Healing

Introduction

Testing shouldn’t be complicated. You know what your application should do — why spend hours writing boilerplate test code?

I built cypress-natural-language-tests to bridge the gap between your test ideas and working Cypress code. Just describe your test in plain English:

python qa_automation.py "Test user login with valid credentials" --run

Result: A complete .cy.js file generated and executed automatically!

And now, with the latest update, the framework also supports Cypress’s new cy.prompt() command for self-healing, AI-powered test execution.

What’s New: cy.prompt() Integration

Cypress recently launched cy.prompt() — their official AI command that converts natural language into test steps at runtime. My framework now supports both approaches:

Mode Description Best For Generate Mode Creates complete .cy.js test files Version control, CI/CD pipelines cy.prompt() Mode Generates tests using cy.prompt() syntax Self-healing tests, rapid prototyping

You choose what works best for your workflow!

How It Works

👆 The complete workflow — from requirements to executed tests

The framework supports two execution paths :

🖥️ Local Machine Flow v/s ⚙️ GitHub Actions CI Flow

🖥️ Local Machine Flow v/s ⚙️ GitHub Actions CI Flow

Two Powerful Modes

Mode 1: Traditional Test Generation

Generate standard Cypress test files that you own and version control:

python qa_automation.py "Test user login with valid credentials"

Output: 01_test-user-login_20241223_102030.cy.js

describe('User Login', () => {
  it('should login successfully with valid credentials', () => {
    cy.visit('https://the-internet.herokuapp.com/login');
    cy.get('#username').type('tomsmith');
    cy.get('#password').type('SuperSecretPassword!');
    cy.get('button[type="submit"]').click();
    cy.get('.flash.success').should('be.visible');
  });
});

Mode 2: cy.prompt() Generation

Generate tests using Cypress’s new AI-powered cy.prompt() command for self-healing capabilities:

python qa_automation.py "Test user login" --use-cyprompt

Output: 01_test-user-login_20241223_102030.cy.js

describe('User Login', () => {
  it('should login successfully with valid credentials', () => {
    cy.prompt([
      'Visit the login page at https://the-internet.herokuapp.com/login',
      'Type "tomsmith" in the username field',
      'Type "SuperSecretPassword!" in the password field',
      'Click the login button',
      'Verify the success message is visible'
    ]);
  });
});

Why cy.prompt()?

🔄 Self-healing : Tests adapt when UI changes

📝 Readable : Natural language steps in your test files

🛡️ Resilient : Less maintenance when selectors change

Quick Start

Installation

# Clone the repository
git clone https://github.com/aiqualitylab/cypress-natural-language-tests.git
cd cypress-natural-language-tests

# Set up Python environment
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txt

# Configure OpenAI API key
echo "OPENAI_API_KEY=your_key_here" > .env

# Initialize Cypress
npm install cypress --save-dev
npx cypress open

Generate Your First Test

# Standard Cypress test
python qa_automation.py "Test user registration flow"

# With cy.prompt() syntax
python qa_automation.py "Test user registration flow" --use -cyprompt

# Generate and run immediately
python qa_automation.py "Test homepage loads correctly" --run

Practical Examples

Example 1: Multiple Test Requirements

python qa_automation.py \
  "Test successful login with valid credentials" \
  "Test login fails with wrong password" \
  "Test login form shows validation errors for empty fields"

Creates three separate test files — one for each requirement.

Example 2: With Documentation Context (RAG)

Supercharge test generation with your own documentation:

python qa_automation.py \
  "Test checkout API according to specifications" \
  --docs ./api-documentation \
  --persist-vstore

The framework indexes your docs into ChromaDB and uses them as context for more accurate test generation.

Example 3: Generate and Execute Locally

python qa_automation.py "Test user profile update" --run

Generates the test AND runs Cypress immediately. View results in your terminal.

Example 4: CI/CD Integration

Trigger via GitHub Actions to generate tests in your pipeline:

- name: Generate Tests
  run: python qa_automation.py "${{ github.event.inputs.requirement }}"

- name: Run Cypress
  run: npx cypress run

- name: Upload Artifacts
  uses: actions/upload-artifact@v3
  with:
    name: cypress-results
    path: |
      cypress/videos
      cypress/screenshots

Why Choose This Framework?

Feature Benefit Dual Mode Support Standard Cypress OR cy.prompt() — your choice Complete Test Files Version control your generated tests Documentation-Aware RAG integration for accurate, context-rich tests Local & CI Ready Works on your machine and in GitHub Actions Model Flexibility Use GPT-4, GPT-4o-mini, or GPT-3.5-turbo Open Source Full control, no vendor lock-in

Configuration

Change AI Model

In qa_automation.py:

llm = ChatOpenAI(
    model="gpt-4o-mini", # Options: gpt-4, gpt-4o, gpt-3.5-turbo
    temperature=0
)

Set Your Application URL

Update the prompt template to target your application:

CY_PROMPT_TEMPLATE = """
...
- Use `cy.visit('https://your-app-url.com')` as the base URL.
...
"""

Get Started Now

🔗 github.com/aiqualitylab/cypress-natural-language-tests

git clone https://github.com/aiqualitylab/cypress-natural-language-tests.git

⭐ Star the repo if you find it useful!

Conclusion

Natural language test generation is here to stay. With cypress-natural-language-tests , you get:

Two modes — Traditional Cypress or cy.prompt()

Full ownership — Complete test files you control

CI/CD ready — Works locally and in GitHub Actions

Documentation-aware — RAG for accurate test generation

Open source — No vendor lock-in

Stop writing boilerplate. Start describing tests in plain English.

What’s your experience with AI-powered test generation? Drop a comment below!

AI-Powered Cypress Test Automation: Automated Test Creation and Execution with Machine Learning

Let's Automate 🛡️ — Fri, 26 Dec 2025 13:56:41 +0000

How to Build Intelligent End-to-End Testing with OpenAI GPT-4, LangChain, LangGraph, and Continuous Integration Pipeline

AI-Powered Cypress Test Automation: Automated Test Creation and Execution with Machine Learning

Transform natural language requirements into production-ready automated tests using OpenAI, LangChain, artificial intelligence and test automation best practices

COMPLETE WORKFLOW

The Problem That Started It All

As a QA engineer specializing in test automation, I’ve spent countless hours writing Cypress tests for web application testing. The manual test creation process was always the same: understand the requirement, inspect the DOM, find the right selectors, write the test code, handle edge cases, and repeat. A simple login test could take 30 minutes. Complex user flows? Hours.

One day, after spending three hours writing automated tests for a basic checkout flow, I thought: “What if I could use artificial intelligence and machine learning to automatically generate test scripts from plain English requirements?”

That question led to building an open-source AI-powered test automation framework that does exactly that — combining natural language processing, automated test generation, and continuous integration for intelligent software testing.

What I Built: An Intelligent Test Automation Framework

The AI-powered testing framework accepts natural language requirements and generates production-ready Cypress E2E tests automatically using machine learning. This automated testing solution combines GPT-4 artificial intelligence with DevOps best practices for continuous testing. Here’s what the intelligent test automation looks like in action:

FULL WORKFLOW

Input:

python qa_automation.py \
  "Test user login with valid credentials" \
  "Test login fails with invalid password" \
  --run

Output:

// 01_test-user-login-with-valid-credentials_20241221_120000.cy.js
describe('User Login', () => {
  it('should login successfully with valid credentials', () => {
    cy.visit('https://the-internet.herokuapp.com/login');
    cy.get('#username').type('tomsmith');
    cy.get('#password').type('SuperSecretPassword!');
    cy.get('button[type="submit"]').click();
    cy.get('.flash.success').should('contain', 'You logged into a secure area!');
  });

  it('should show error with invalid credentials', () => {
    cy.visit('https://the-internet.herokuapp.com/login');
    cy.get('#username').type('invaliduser');
    cy.get('#password').type('wrongpassword');
    cy.get('button[type="submit"]').click();
    cy.get('.flash.error').should('contain', 'Your username is invalid!');
  });
});

The framework works both locally and in CI/CD pipelines, generating tests in seconds instead of hours.

LOCAL FLOW

The Technical Architecture

Core Components

The system consists of four main pieces:

1. Python Orchestration Layer I built the core in Python, using LangGraph to manage the workflow. LangGraph provides a graph-based state management system perfect for orchestrating complex AI workflows.

2. OpenAI Integration The heart of the system uses GPT-4o-mini. I chose this model for its balance of speed, cost-effectiveness, and code generation quality.

3. Cypress Test Runner The generated tests are standard Cypress JavaScript files that run without modification in any Cypress environment.

4. Optional Context Store Using ChromaDB, the framework can index project documentation to provide additional context for more accurate test generation.

How It Works Internally

Here’s the step-by-step process:

Step 1: Requirement Parsing

def parse_cli_args(state: QAState) -> QAState:
    parser = argparse.ArgumentParser(
        description="Generate Cypress tests from natural language"
    )
    parser.add_argument("requirements", nargs="+")
    parser.add_argument("--run", action="store_true")
    args = parser.parse_args()
    state["requirements"] = args.requirements
    return state

Step 2: AI Generation I crafted a prompt template that guides GPT-4 to generate Cypress-compliant code:

CY_PROMPT_TEMPLATE = """You are a senior automation engineer.
Write a Cypress test for: {requirement}

Constraints:
- Use Cypress best practices
- Include describe and it blocks
- Use real selectors (id, class, name)
- Include positive and negative test paths
- Return ONLY runnable JavaScript code
"""

Step 3: Code Generation and Validation The LLM returns raw JavaScript code, which I save with descriptive filenames:

def generate_tests(state: QAState) -> QAState:
    for idx, req in enumerate(state["requirements"], start=1):
        code = generate_cypress_test(req)
        slug = slugify(req)[:60]
        filename = f"{idx:02d}_{slug}_{now_stamp()}.cy.js"
        filepath = Path(out_dir) / filename
        with open(filepath, "w") as f:
            f.write(f"// Requirement: {req}\n")
            f.write(code)

Step 4: Optional Execution If the --run flag is provided, the framework executes Cypress immediately:

def run_cypress(state: QAState) -> QAState:
    if state.get("run_cypress"):
        specs = state.get("generated_files", [])
        subprocess.run(["npx", "cypress", "run", "--spec", ",".join(specs)])

The Workflow

LangGraph enabled me to build a clean, maintainable workflow. Here’s the graph structure:

def create_workflow():
    graph = StateGraph(QAState)
    graph.add_node("ParseCLI", parse_cli_args)
    graph.add_node("BuildVectorStore", create_or_update_vector_store)
    graph.add_node("GenerateTests", generate_tests)
    graph.add_node("RunCypress", run_cypress)

    graph.set_entry_point("ParseCLI")
    graph.add_edge("ParseCLI", "BuildVectorStore")
    graph.add_edge("BuildVectorStore", "GenerateTests")
    graph.add_edge("GenerateTests", "RunCypress")
    graph.add_edge("RunCypress", END)

    return graph.compile()

This graph-based approach makes it easy to add new nodes (like validation, reporting, or test optimization) without refactoring the entire codebase.

CI/CD Integration

The framework shines in automated environments. I built a GitHub Actions workflow that:

CI/CD

Accepts test requirements as workflow inputs

Sets up Node.js and Python environments

Generates tests using AI

Executes them with Cypress

Uploads videos, screenshots, and test files as artifacts

The workflow file looks like this:

name: AI-Powered Cypress Tests
on:
  push:
  pull_request:
  workflow_dispatch:
    inputs:
      requirements:
        description: 'Test requirements (one per line)'
        required: true
jobs:
  generate-and-run-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20.x'

      - name: Setup Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'

      - name: Install dependencies
        run: |
          npm install
          pip install -r requirements.txt

      - name: Generate and run tests
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python qa_automation.py \
            "Test login functionality" \
            "Test checkout process" \
            --run --out cypress/e2e/generated

Challenges and Solutions

Challenge 1: Selector Discovery

Problem: How does the AI know what selectors exist on the page?

Solution: I refined the prompt to instruct the model to use common, semantic selectors. For better accuracy, I added an optional documentation context feature using ChromaDB:

def create_or_update_vector_store(state: QAState):
    docs_dir = state.get("docs_dir")
    if docs_dir:
        loader = DirectoryLoader(docs_dir, glob="**/*.*")
        documents = loader.load()
        splitter = RecursiveCharacterTextSplitter(chunk_size=800)
        chunks = splitter.split_documents(documents)
        db = Chroma.from_documents(chunks, embeddings, 
                                    persist_directory=VECTOR_STORE_DIR)

This allows users to provide API documentation or page structure files for more accurate selector generation.

Challenge 2: Test Quality Consistency

Problem: LLM outputs can vary in quality.

Solution: I implemented strict prompt engineering:

Explicit instructions for Cypress best practices

Requirement to include both positive and negative test cases

Mandate for clear, descriptive assertions

Instruction to return only executable JavaScript (no explanations)

Challenge 3: Handling Multiple Requirements

Problem: Processing requirements sequentially was slow.

Solution: While I kept sequential processing for simplicity and cost control, the architecture supports parallel processing. Each requirement is independent, making it trivial to parallelize in the future:

# Future enhancement potential
from concurrent.futures import ThreadPoolExecutor
def generate_tests_parallel(state: QAState):
    with ThreadPoolExecutor(max_workers=5) as executor:
        futures = [executor.submit(generate_cypress_test, req) 
                   for req in state["requirements"]]
        results = [f.result() for f in futures]

Real-World Usage Examples

Example 1: E-commerce Testing

python qa_automation.py \
  "Test product search returns relevant results" \
  "Test adding multiple items to cart" \
  "Test checkout with valid payment information" \
  "Test order confirmation email is sent" \
  --run

Example 2: User Authentication Flows

python qa_automation.py \
  "Test user registration with valid email" \
  "Test registration fails with existing email" \
  "Test login with correct credentials" \
  "Test password reset flow" \
  "Test account lockout after failed attempts" \
  --run

Example 3: Form Validation

python qa_automation.py \
  "Test contact form with all fields filled correctly" \
  "Test form shows errors for empty required fields" \
  "Test email validation rejects invalid formats" \
  "Test phone number accepts international formats" \
  --run

Measurable Impact

After using this framework for several projects:

Time savings: 95% reduction in test writing time (30 minutes → 90 seconds per test)

Test coverage: Ability to generate 50+ tests in the time it previously took to write 2–3

Maintenance: Regenerating tests for UI changes takes seconds instead of hours

Onboarding: New team members can contribute tests on day one without Cypress expertise

Getting Started

The framework is open source and available on GitHub. Here’s how to set it up:

Installation:

git clone https://github.com/aiqualitylab/cypress-natural-language-tests
cd cypress-natural-language-tests
npm install
pip install -r requirements.txt

Configuration:

# Create .env file
echo "OPENAI_API_KEY=your_key_here" > .env

# Create cypress.config.js
cat > cypress.config.js << 'EOF'
const { defineConfig } = require('cypress')
module.exports = defineConfig({
  e2e: {
    baseUrl: 'https://your-app.com',
    supportFile: false,
    video: true,
    screenshotOnRunFailure: true,
  },
})
EOF

Usage:

# Generate and run tests
python qa_automation.py \
  "Your test requirement here" \
  --run

# Generate only (no execution)
python qa_automation.py \
  "Your test requirement here"

Lessons Learned

On Prompt Engineering

The quality of generated tests is directly proportional to prompt quality. I spent significant time iterating on the prompt template, testing with various requirement phrasings.

On LLM Selection

GPT-4o-mini proved to be the sweet spot for this use case. GPT-3.5 was too inconsistent, while full GPT-4 was unnecessarily expensive for test generation.

On Workflow Design

LangGraph’s state-based approach simplified complex orchestration. The ability to visualize the workflow graph helped identify bottlenecks and optimization opportunities.

On Integration

Making the framework work seamlessly in both local and CI/CD environments required thoughtful design. The key was keeping the core logic environment-agnostic and using configuration for environment-specific behavior.

Conclusion: The Future of Intelligent Test Automation

Building this AI-powered test automation framework transformed how I approach software testing and quality assurance. What once took hours now takes seconds with automated test generation. What required deep Cypress expertise now requires clear requirement writing using natural language processing.

This intelligent testing framework isn’t just about speed — it’s about democratizing test automation and making QA accessible. Anyone who can describe what should be tested can now generate automated tests, regardless of their programming background, thanks to machine learning and artificial intelligence.

The code is open source, the CI/CD workflow is extensible, and the potential applications go far beyond Cypress test automation. From end-to-end testing to integration testing, this AI-driven approach represents the future of software quality assurance. I’m excited to see how the DevOps and testing community builds upon this foundation for intelligent test automation.

Try It Yourself

GitHub Repository: https://github.com/aiqualitylab/ai-natural-language-tests

Documentation: See the README for detailed setup and usage instructions

Issues/Contributions: Pull requests and feature suggestions welcome!

Connect With Me

I’m passionate about AI-powered quality engineering and love discussing test automation innovations. Find me on:

GitHub: @aiqualitylab

Medium: Follow for more articles on AI and testing

What would you build with AI-generated tests? Share your ideas in the comments below!

Appendix: Complete Code Example

Here’s a simplified version of the core generation function:

import os
from langchain_openai import ChatOpenAI
from dotenv import load_dotenv

load_dotenv()
def generate_cypress_test(requirement: str) -> str:
    """Generate Cypress test code from natural language requirement"""

    llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

    prompt = f"""You are a senior automation engineer.
Write a Cypress test in JavaScript for: {requirement}
Requirements:
- Use Cypress best practices
- Include describe and it blocks  
- Use real page selectors
- Include positive and negative paths
- Return ONLY runnable JavaScript code
Code:"""

    result = llm.invoke(prompt)
    return result.content.strip()
# Example usage
test_code = generate_cypress_test("Test user login with valid credentials")
print(test_code)

This example demonstrates the core concept. The full framework adds error handling, state management, file organization, and CI/CD integration.

Thank you for reading! If you found this helpful, please give it a clap 👏 and share with others who might benefit from AI-powered test automation.

GitHub Copilot Agent Skills: Teaching AI Your Repository Patterns

Let's Automate 🛡️ — Sat, 20 Dec 2025 18:06:36 +0000

A practical guide to the new GitHub Copilot Agent Skills feature (announced December 18, 2025)

Example Repository: SeleniumSelfHealing.Reqnroll

Every test automation engineer faces this challenge: you build a sophisticated framework with custom patterns, then AI assistants suggest brittle, outdated approaches that ignore your architecture.

On December 18, 2025, GitHub announced Agent Skills — folders containing instructions, scripts, and resources that Copilot automatically loads when relevant to your prompt. This feature works across the Copilot coding agent, Copilot CLI, and agent mode in Visual Studio Code.

Let me show you how I used this to teach Copilot our self-healing Selenium patterns.

What Are Agent Skills?

According to GitHub's announcement, Agent Skills allow you to teach Copilot how to perform specialized tasks in a specific, repeatable way. When Copilot determines a skill is relevant to your task, it loads the instructions and follows them.

You create skills by adding a .github/skills/[skill-name]/SKILLS.md file to your repository. The skills work automatically—no manual activation needed.

The Problem: Brittle Selenium Tests

Traditional Selenium tests break easily:

// Typical brittle approach
var button = driver.FindElement(By.XPath("//button[@id='submit-2023']"));
button.Click();

When element IDs change, tests fail. Our framework uses AI-powered element recovery with semantic descriptions instead of hardcoded selectors. But without guidance, Copilot suggested the old brittle patterns.

Creating the Agent Skill

Here's the structure I implemented:

# Selenium Self-Healing Automation Skills

## Purpose
Enable Copilot to:
- Generate robust Selenium UI tests
- Use AI-powered self-healing locator strategies
- Follow BDD patterns with Reqnroll

## Hard Rules

### Must
- Use self-healing WebDriver extensions
- Prefer element descriptions over raw locators
- Generate async step definitions
- Log all healing attempts

### Must Not
- Hardcode XPath or CSS selectors
- Use Thread.Sleep
- Bypass self-healing logic

## Golden Example
Step Definition Pattern:

[When(@"I click the ""(.*)""")]
public async Task WhenIClickElement(string elementDescription)
{
    await _driver.Click(
        By.CssSelector(""),
        elementDescription
    );
}

Scenario: Search for Selenium
  Given I navigate to "https://www.wikipedia.org"
  When I enter "Selenium" into the "search box"
  And I click the "search button"
  Then I should see "Selenium"

The Results

After adding the skill, developers type:

// Create step definition to click login button

Copilot now generates:

[When(@"I click the ""(.*)""")]
public async Task WhenIClickElement(string elementDescription)
{
    await _driver.Click(By.CssSelector(""), elementDescription);
}

Perfect! It follows our self-healing pattern with semantic descriptions instead of brittle locators.

Key Components That Work

Clear Rules Define explicit must-do and must-not-do items. Specificity produces better results.

Working Examples Use actual code from your repository. Copilot learns from real patterns, not theoretical ones.

Context About Structure Explain your project organization so Copilot places code correctly.

Templates Provide scaffolding for scenarios developers encounter frequently.

Verifying It Works

According to GitHub's documentation, when Copilot chooses to use a skill, the SKILL.md file will be injected in the agent's context.

To verify:

Open Copilot Chat and ask: "How should I create a new step definition?"
Look for references to your SKILLS.md in the response
Check suggestions match your patterns (async methods, element descriptions, self-healing extensions)

Create a test file with a comment triggering your patterns and observe what Copilot suggests.

Important Requirements

GitHub Copilot Agent Skills requires a paid plan (Individual, Business, or Enterprise). The feature is available with Copilot coding agent, GitHub Copilot CLI, and agent mode in Visual Studio Code Insiders. Support in the stable version of VS Code is coming soon.

Without a paid plan, the SKILLS.md still serves as valuable documentation for your team.

The feature may need 5–10 minutes to index new files. Reload your IDE if suggestions don't immediately reflect your patterns.

Why This Matters for Test Automation

Testing frameworks evolve beyond standard practices. Self-healing locators, AI-powered recovery, custom assertions — these patterns don't exist in Copilot's base training.

GitHub notes that you can write your own skills, or use skills shared by others, such as those in the anthropics/skills repository or GitHub's community created github/awesome-copilot collection.

This transforms AI from suggesting generic approaches to understanding your specific methodology. It's not about speed — it's about generating the right code that maintains your architecture's quality standards.

Getting Started

Create .github/skills/your-skill/SKILLS.md
Document your most critical pattern
Include one golden example
Test with a new file
Expand as you identify more patterns

Start small. Focus on the pattern that matters most. You can expand later.

Currently, skills can only be created at the repository level. Support for organization-level and enterprise-level skills is coming soon.

Additional Resources

GitHub Changelog: GitHub Copilot now supports Agent Skills
Documentation: About Agent Skills — GitHub Docs
Community Skills: github/awesome-copilot

What patterns would you teach GitHub Copilot in your test automation projects?

The Complete Guide to Testing Types: Traditional vs AI Era

Let's Automate 🛡️ — Thu, 18 Dec 2025 19:18:00 +0000

The Complete Guide to Testing Types: Traditional vs AI Era

As someone deep in the AI-powered testing space, I've noticed a fascinating evolution happening. We're not replacing traditional testing - we're expanding our toolkit. Let me break down both worlds for you.

The Testing Landscape: A Visual Map

Part 1: Traditional Testing - The Foundation

Functional Testing: Does It Work?

This is where most QA engineers start. You're verifying the software behaves as expected.

Unit Testing - Think of this as testing individual LEGO blocks before building the castle. Each function or method gets its own test suite.

Integration Testing - Now we're connecting those LEGO blocks. Does the login module talk to the database correctly? Does the payment gateway integrate with the order system?

System Testing - The entire castle is built. We're testing the whole application end-to-end in an environment that mirrors production.

Acceptance Testing - This is where business stakeholders say "Yes, this meets our needs." Often called UAT (User Acceptance Testing).

Regression Testing - After adding new features, we verify nothing broke. This is where automation shines!

Smoke Testing - Quick sanity checks after deployment. "Is the application even running?"

Sanity Testing - More focused than smoke tests. After a bug fix, we verify that specific area works without retesting everything.

Non-Functional Testing: How Well Does It Work?

Performance Testing is my favorite category because it reveals how your app behaves under real-world conditions.

Load Testing: Simulating expected user traffic. Can your app handle 10,000 concurrent users?
Stress Testing: Pushing beyond normal capacity. What's the breaking point?
Spike Testing: Sudden traffic surges (think Black Friday sales). Does your system gracefully handle it?

Security Testing - Finding vulnerabilities before hackers do. SQL injection, XSS, authentication flaws.

Usability Testing - Can users actually navigate your interface? This often gets overlooked by developers.

Compatibility Testing - Testing across browsers, devices, OS versions. Mobile vs desktop experiences.

Reliability Testing - Can your system run continuously without failure? Mean time between failures (MTBF) matters.

Structural Testing: The Perspective Matters

White Box Testing - You see the code. You're testing internal logic, code paths, and structure.

Black Box Testing - You're testing like an end-user. No knowledge of how it's implemented.

Gray Box Testing - Best of both worlds. Partial knowledge helps design better test cases.

Part 2: AI/ML Testing - The New Frontier

Here's where things get interesting. AI systems are fundamentally different from traditional software.

Data Testing: Garbage In, Garbage Out

AI models are only as good as their training data. Data testing becomes critical.

Data Quality Testing - Are your datasets complete? Accurate? Consistent? Missing values? Duplicates?

Data Validation - Checking schemas, data types, value ranges, statistical distributions. If your model expects images at 224x224 but gets 100x100, things break.

Data Drift Testing - Production data often differs from training data over time. User behavior changes. New edge cases emerge. Monitoring drift prevents model degradation.

Model Testing: Beyond Accuracy

Model Accuracy Testing - Measuring precision, recall, F1-score, AUC-ROC. But accuracy alone isn't enough.

Model Performance Testing - Inference latency matters. A 99% accurate model that takes 10 seconds per prediction is useless in real-time systems.

Robustness Testing - How does your model handle edge cases? Noisy input? Adversarial examples? Missing features?

Metamorphic Testing - Here's a clever technique: apply transformations that shouldn't change the outcome. Rotating an image of a cat should still classify it as a cat.

AI System Testing: Production Reality

Integration Testing - How does your ML model integrate with APIs, databases, frontend applications?

End-to-End Testing - Testing complete workflows. User submits a photo → Model processes → Results displayed → Action taken.

A/B Testing - Running two model versions simultaneously to compare performance. Model v2 might be more accurate but slower.

Shadow Testing - Running new models alongside production without affecting users. Comparing predictions to validate before full deployment.

Ethical & Bias Testing: The Responsibility Factor

This is where AI testing diverges significantly from traditional testing.

Bias Testing - Does your hiring algorithm discriminate based on gender? Does your loan approval model have racial bias?

Fairness Testing - Ensuring equitable outcomes across demographic groups. Statistical parity, equal opportunity, individual fairness.

Explainability Testing - Can you explain why the model made a decision? Critical for regulated industries (healthcare, finance, legal).

Adversarial Testing - Intentionally crafting inputs to fool your model. Adding noise to images, manipulating text, poisoning data.

The Fundamental Shift

Traditional software is deterministic. Same input → Same output. Every time.

AI systems are probabilistic. Same input → Potentially different outputs. Statistical validation required.

This means:

Test assertions become threshold-based ("accuracy > 95%") rather than exact matches
Continuous monitoring replaces point-in-time testing
Data pipelines need as much testing as code
Model versioning and rollback strategies become critical
Ethical considerations join functional requirements

Practical Implications for QA Engineers

If you're coming from traditional QA like I did, here's what changes:

Learn statistics: You'll need to understand confusion matrices, ROC curves, statistical significance.
Data engineering skills: SQL, data pipelines, feature engineering become part of your toolkit.
Domain knowledge matters more: Understanding what "good" looks like for a medical diagnosis model requires healthcare knowledge.
Testing becomes ongoing: Models degrade over time. Monitoring isn't optional.
Collaborate differently: You'll work closely with data scientists, ML engineers, domain experts.

Tools of the Trade

Traditional Testing: Selenium, Playwright, JUnit, NUnit, Postman, JMeter, Cypress

AI Testing: Great Expectations, MLflow, Evidently AI, DeepChecks, Weights & Biases, TensorBoard

Bridging Both: That's where frameworks like my SeleniumSelfHealing.Reqnroll project come in - using AI to make traditional testing more robust.

Wrapping Up

We're not abandoning traditional testing principles. We're extending them. The fundamentals of good testing - clear objectives, reproducibility, comprehensive coverage - remain vital.

But AI introduces new challenges: non-deterministic behavior, data dependencies, ethical considerations, continuous degradation. Our testing strategies must evolve accordingly.

The future QA engineer needs feet in both worlds. Master traditional testing techniques while embracing AI-specific methodologies. It's an exciting time to be in quality assurance.

What testing types are you working with? Traditional, AI, or both? Drop a comment below!

P.S. If you're interested in AI-powered test automation, check out my open-source projects on GitHub @aiqualitylab or read more on aiqualityengineer.com

Tags: #testing #qa #ai #machinelearning #automation #softwaredevelopment #qualityassurance #devops

Testing AI Systems: Handling the Test Oracle Problem

Let's Automate 🛡️ — Wed, 17 Dec 2025 20:09:54 +0000

AI systems are typically a blend of AI components, such as machine learning models, and non-AI components, like APIs, databases, or UI layers. Testing the non-AI parts of these systems is similar to testing traditional software. Standard techniques like boundary testing, equivalence partitioning, and automation can be applied effectively. However, the AI components present a different set of challenges. Their complexity, unpredictability, and data-driven nature require a specialized approach to testing.

Key Challenge: The Test Oracle Problem

In traditional software testing, we compare the actual results of a test with the expected results, which serve as the "oracle." This comparison determines whether the test has passed or failed. However, in AI systems, defining what the "correct" output should be for every possible input is often difficult. This is known as the "test oracle problem."

This difficulty arises because:

AI behavior is probabilistic, not deterministic: AI models, especially machine learning models, don't always produce the same output for the same input. There's often an element of randomness involved.

Outputs can vary even for similar inputs: Small changes in the input data can sometimes lead to significant changes in the output, making it hard to predict the expected behavior.

Techniques to Tackle the Oracle Problem

Several techniques can be used to address the test oracle problem in AI systems:

Back-to-Back Testing

This technique involves comparing the outputs of two systems performing the same task. One system can serve as a reference for the other.

How it works: You run the same input through both systems and compare their outputs. If the outputs are significantly different, it indicates a potential issue in one of the systems.

Use cases:

Regression testing: Comparing the output of a new version of a model with the output of a previous, trusted version.
Baseline comparison: Comparing the output of a new model with the output of a different model that is known to perform well.

Benefits: Useful when a trusted baseline exists or for detecting regressions in model performance.

A/B Testing

A/B testing involves comparing two versions of a model in a production environment.

How it works: Real users are randomly assigned to one of the two versions of the model. The performance of each version is then measured based on user behavior and feedback.

Use cases:

Self-learning systems: Evaluating the impact of new training data or model updates on real-world performance.
Live model updates: Ensuring that new model versions perform as expected before fully deploying them.

Benefits: Allows for testing with real user input and detecting changes, regressions, or data poisoning in a live environment.

Metamorphic Testing

Metamorphic testing relies on identifying logical relations between inputs and outputs.

How it works: Instead of knowing the exact correct output for a given input, you define relationships that should hold true. For example, if you rotate an image of a cat, the model should still identify it as a cat.

Example: If rotating a cat image still shows "cat," the model is consistent.

Benefits: Helps find issues without knowing the exact correct output. It is especially helpful for non-experts.

Limitations: There are currently no commercial tools available for metamorphic testing; it is mostly a manual process.

Other AI-Specific Testing Techniques

Adversarial Testing

Adversarial testing involves feeding tricky or intentionally misleading inputs to the model.

How it works: You create inputs that are designed to exploit weaknesses in the model and cause it to make incorrect predictions.

Use cases:

Security-sensitive systems: Identifying vulnerabilities that could be exploited by malicious actors.
Safety-critical systems: Ensuring that the model can handle unexpected or unusual inputs without causing harm.

Benefits: Checks robustness and is useful in security-sensitive or safety-critical systems.

Data Poisoning Tests

Data poisoning tests involve injecting bad or malicious data into the training sets to see if the model can be corrupted.

How it works: You introduce flawed or biased data into the training data used to build the model. Then, you observe how the model's performance changes as a result.

Use cases:

AI systems exposed to untrusted or public data sources: Protecting against malicious actors who might try to manipulate the model by injecting bad data.

Benefits: Important for AI systems exposed to untrusted or public data sources.

Pairwise Testing

Pairwise testing involves testing all combinations of input parameter pairs.

How it works: You identify the key input parameters that affect the model's behavior. Then, you create test cases that cover all possible combinations of these parameters.

Benefits: Reduces test set size while covering more interactions in complex models.

Experience-Based Testing

Experience-based testing leverages domain knowledge and tester intuition.

How it works: Testers use their understanding of the system and the data to design test cases that are likely to uncover issues. This often includes Exploratory Data Analysis (EDA) to understand the data used in training.

Benefits: Useful when model behavior depends heavily on the data.

Neural Network Coverage

Neural network coverage is similar to code coverage but applied to neural networks.

How it works: You measure the extent to which the test cases exercise different parts of the neural network.

Benefits: Ensures all parts of the model logic are exercised. Useful for deep learning models to detect untested paths.

Summary for QA Teams

Testing AI components isn't about verifying fixed outputs. It's about understanding behavior, patterns, and risks.

Choosing the right mix of testing techniques depends on:

Risk level: (e.g., safety, security)
System complexity
Data quality
Model type: (static vs. self-learning)

By combining traditional testing with AI-specific methods, QA teams can validate AI systems effectively and ensure they're reliable, safe, and fair.