DEV Community: Shiplight

What Is AI Testing?

Shiplight — Tue, 21 Apr 2026 08:36:22 +0000

"AI testing" has become one of the most-searched terms in software quality. But because the label is broad, it means different things to different tools. Some vendors use "AI testing" to describe smart locators in a Selenium script; others use it to describe fully autonomous QA agents that plan, execute, and heal tests without human intervention. These are not the same thing.

This guide defines AI testing as a category, maps the five subcategories that matter in 2026, explains how each fits into real engineering workflows, and helps you identify which part of the category addresses your specific problem.

What Is AI Testing?

AI testing is the use of artificial intelligence — large language models (LLMs), machine learning, and related techniques — to automate tasks in the software quality assurance lifecycle that were previously manual. Those tasks include:

Deciding what to test
Writing test cases
Executing tests in a real browser or runtime
Interpreting failures and distinguishing real bugs from flakiness
Maintaining tests as the application changes

Traditional test automation (Selenium, Cypress, Playwright scripts) automates only execution — humans still write, interpret, and maintain tests. AI testing automates the other stages, each to different degrees depending on the specific tool and category.

See generative AI in software testing for a deeper look at how generative models specifically are applied, and what is agentic QA testing? for the most autonomous subcategory.

AI Testing vs. Generative AI in Testing

A common confusion: "AI testing" and "generative AI in software testing" overlap but are not identical.

Generative AI in testing is a technique — using LLMs to produce new artifacts (test cases, healing patches, test data). It powers three of the five AI testing categories below. See generative AI in software testing for the full technical breakdown.

AI testing is the broader category — it includes generative AI applications plus rule-based AI features (smart locators, flakiness detection) and non-generative authoring experiences (no-code visual builders, low-code YAML). All five categories below are AI testing; only three are primarily generative.

The 5 Categories of AI Testing in 2026

Generative-AI-powered categories

1. AI Test Generation

AI produces test cases from specs, user stories, or live app exploration — replacing manual authoring. See what is AI test generation? for the deep dive, and AI testing tools that automatically generate test cases for the tool comparison.

2. Self-Healing Test Automation

AI repairs tests when the UI changes, using either locator fallback or intent-based re-resolution. See what is self-healing test automation? and best self-healing test automation tools.

3. Agentic QA

AI agents handle the full quality lifecycle autonomously — the most autonomous subcategory. See what is agentic QA testing?, best agentic QA tools in 2026, and agent-native autonomous QA.

Non-generative AI categories

4. AI-Augmented Automation

AI-augmented automation adds rule-based AI features — smart locators, flakiness detection, visual diff scoring, assisted authoring — to fundamentally script-based frameworks. Unlike generative AI, these features don't produce new artifacts. They improve existing tests by making selectors more robust, execution more stable, or failures more actionable.

Typical AI-augmented features:

Smart locators — the tool watches which attributes of an element are stable and automatically prefers those over brittle CSS selectors or XPath. Unlike intent-based healing, this is deterministic pattern matching, not semantic re-resolution.
Flakiness detection — statistical analysis of test history identifies tests that pass or fail intermittently, flagging them for investigation. See how to fix flaky tests and flaky tests to actionable signal.
Visual diff scoring — AI ranks the significance of pixel differences between screenshots, reducing false positives in visual regression testing.
Assisted authoring — AI suggests the next test step based on user interactions or spec context, but the engineer still writes the test.

Tools that fit this category: Katalon's AI features, Tricentis Testim, Mabl's auto-wait and healing, Applitools' visual AI. Most "AI-powered" marketing from legacy test automation vendors refers to this category, not to the more ambitious generative or agentic categories.

Where this category fits: Teams with existing script-based test suites who want to reduce flakiness and maintenance burden without rewriting their entire approach. The ROI is incremental improvement, not transformation.

5. No-Code Testing

No-code testing is an authoring model where tests are created through visual builders, plain-English sentences, YAML with natural-language intent, or record-and-playback — without writing code. It is orthogonal to the AI technique being used: a no-code tool might use generative AI under the hood, or rule-based logic, or pure interpretation of recorded actions.

What makes no-code testing a distinct AI testing category is who creates tests, not how the AI works. When authoring is accessible to non-engineers — product managers, designers, QA analysts, business users — a different operating model becomes possible:

Specifications become tests directly — the person who defines product behavior can encode that behavior as a test, eliminating translation loss from PM → engineer → test
Review happens in plain language — PMs can approve tests as readable specifications, not as code they don't understand
Coverage broadens — the testing team effectively grows beyond engineering headcount

No-code testing exists on a spectrum:

Pure no-code — zero code, zero structured markup (testRigor plain English)
Low-code — structured format with optional code extensions (Shiplight YAML, Mabl visual)
Record-and-playback — generated from user interactions (codeless E2E testing)

See what is no-code test automation? for the conceptual foundation, best no-code test automation platforms and best low-code test automation tools for tool roundups, and no-code testing for non-technical teams for the adoption guide.

Where this category fits: Teams where QA is owned by non-engineers, or teams that want product managers and designers to contribute to test coverage without learning a programming language.

Quick Category Comparison

Category	Automates	Human role	Best for
AI test generation	Authoring	Review generated tests	Teams that can't write tests fast enough
Self-healing	Maintenance	Review healing patches	Teams whose tests break constantly on UI changes
Agentic QA	Full lifecycle	Oversight and policy	Teams with AI coding agents, high velocity
AI-augmented	Parts of authoring + maintenance	Write tests; AI helps	Teams with existing scripted suites
No-code	Authoring for non-engineers	Specify intent	Teams where QA is owned by non-engineers

Most teams adopt a combination. See best AI testing tools in 2026 for a tool-by-tool breakdown across all categories, or best AI automation tools for software testing for a broader category roundup.

How AI Testing Differs from Traditional Test Automation

Traditional test automation with Playwright, Selenium, or Cypress automates execution only. Humans still:

Decide what to test (manual planning)
Write test code targeting specific selectors (manual authoring)
Run the tests (automated, but triggered manually or in CI)
Diagnose failures (manual — is this a real bug or a broken test?)
Fix broken selectors when the UI changes (manual maintenance)

AI testing automates steps 1, 2, 4, and 5 to varying degrees depending on the subcategory. Fully agentic QA automates all five; self-healing tools focus on step 5; AI test generation focuses on steps 1 and 2.

The practical effect: AI testing scales with development velocity rather than against it. When AI coding agents like Claude Code, Cursor, Codex, and GitHub Copilot produce code faster than humans can write tests for it, traditional automation falls behind. AI testing keeps up.

Benefits of AI Testing

Coverage scales with development velocity

Manual authoring is the bottleneck when AI coding agents produce code at machine speed. AI testing removes that bottleneck.

Tests survive UI changes

Self-healing, especially intent-based healing, means tests don't break every sprint — they adapt automatically.

Non-engineers can contribute

No-code and natural-language authoring open testing to product managers, designers, and QA analysts who previously couldn't write tests.

Integration with AI coding agents

Tools like Shiplight Plugin expose testing as Model Context Protocol (MCP) capabilities the coding agent can call during development — closing the loop between AI code generation and AI quality verification.

Fast time-to-coverage

AI-generated tests cover new features in minutes rather than days of manual authoring.

Limitations of AI Testing

Hallucinated tests

LLMs sometimes generate tests for behavior that doesn't exist or with incorrect expected values. Human review remains necessary, particularly for business-rule-heavy flows.

Opaque failure modes

When AI systems fail, the reasoning is often not inspectable. This creates debugging friction and compliance concerns in regulated industries.

Data residency

Generative AI tools typically send application state and DOM content to LLM providers. This creates security and compliance considerations not present with self-hosted frameworks.

Not a replacement for every test type

AI testing excels at UI-level E2E. Unit tests, integration tests, performance tests, and many security tests remain better served by specialized tools.

How to Adopt AI Testing

Step 1: Identify your primary bottleneck

If your pain is…	Start with…
Writing new tests takes too long	AI test generation
Tests break constantly when UI changes	Self-healing test automation
AI coding agents ship untested code	Agentic QA with MCP integration
Fixture data is stale or unrealistic	Test data generation (part of AI test generation)
QA is a release-cadence bottleneck	Agentic QA
Non-engineers need to contribute	No-code testing

Step 2: Run a 30-day pilot

Pick one high-value user flow. Implement it fully with the AI testing category you chose. Measure: time to first test, healing success rate on intentional UI changes, and failure signal quality.

Step 3: Expand by coverage, not by tool

Add more flows using the same tool before adding additional AI testing categories. Vertical depth first, horizontal breadth second.

Step 4: Establish governance

Define who reviews AI outputs, how test changes flow through code review, and what data leaves your environment. For regulated industries, see best self-healing test automation tools for enterprises.

FAQ

What is AI testing?

AI testing is the use of artificial intelligence — large language models, machine learning, and related techniques — to automate tasks in software quality assurance that were previously manual. It spans five categories: AI test generation, self-healing test automation, agentic QA, AI-augmented automation, and no-code testing. Each category automates a different part of the testing lifecycle.

Is AI testing the same as test automation?

No. Traditional test automation (Playwright, Selenium, Cypress) automates test execution — humans still write, interpret, and maintain the tests. AI testing automates the other stages: authoring, interpretation, and maintenance, to varying degrees depending on the subcategory.

What are the types of AI testing?

Five distinct categories: AI test generation (AI creates tests from specs or exploration), self-healing test automation (tests repair themselves when UIs change), agentic QA (AI handles the full testing lifecycle autonomously), AI-augmented automation (AI features added to script-based frameworks), and no-code testing (AI enables non-engineers to author tests through visual or natural-language interfaces).

Can AI testing replace human QA engineers?

No — it replaces execution work, not judgment work. AI testing handles authoring, maintenance, execution, and triage. Human QA engineers shift to setting quality policy, reviewing edge cases, and handling domain-specific judgment calls. Teams typically see QA headcount stabilize while coverage grows, not decrease.

Is AI testing production-ready in 2026?

Yes for most categories. Self-healing, AI test generation, and agentic QA are in production at teams ranging from AI-native startups to enterprises. AI coding agent verification via Shiplight Plugin is newer but production-ready with SOC 2 Type II certification. Fully autonomous test interpretation without any human review is still emerging.

How does AI testing fit with AI coding agents like Claude Code or Cursor?

AI coding agents generate code; AI testing verifies it. The integration point is Model Context Protocol (MCP) — agentic QA tools like Shiplight expose testing capabilities as MCP tools the coding agent can call during development, closing the loop between AI code generation and AI quality verification. See agent-native autonomous QA for the full paradigm.

What's the difference between AI testing and AI-powered testing?

Usually used interchangeably, but "AI-powered" is often marketing shorthand from vendors adding minor AI features to otherwise traditional tools. "AI testing" in its substantive form covers all five categories above — not just smart locators on a Selenium script.

Conclusion

AI testing is not one thing — it is five distinct categories, each at different levels of maturity. The highest-leverage adoption path depends on where your team's bottleneck is: authoring, maintenance, coverage, or integration with AI coding agents.

For teams building with AI coding agents, Shiplight AI spans all five categories in one platform: AI test generation, intent-based self-healing, agentic QA, AI coding agent verification via MCP, and no-code YAML authoring readable by non-engineers. Tests live in your git repository, survive UI changes, and run in any CI environment.

Get started with Shiplight Plugin.

Best Low-Code Test Automation Tools in 2026: 7 Platforms Compared

Shiplight — Tue, 21 Apr 2026 02:22:02 +0000

Originally published on the Shiplight blog.

The best low-code test automation tools in 2026 are Shiplight AI (intent-based YAML with AI coding agent integration), Mabl (visual builder with auto-healing), Katalon (record-and-playback plus scripting), testRigor (plain-English authoring), ACCELQ (codeless cross-platform), Functionize (ML-driven NLP), and Virtuoso QA (natural language with visual testing).

"Low-code test automation" sits in the middle of a spectrum — more structured than purely no-code plain-English tools, less code-intensive than frameworks like Playwright or Selenium. It has become the dominant authoring model for modern testing platforms because it lets engineers and non-engineers both contribute to the same test suite.

In 2026, seven low-code test automation tools dominate the category. They differ in authoring format, self-healing quality, AI coding agent support, and enterprise readiness. We build Shiplight AI, so it's listed first — but we'll be honest about where each alternative excels.

What Is Low-Code Test Automation?

No-code — zero code at any stage (testRigor plain English)
Code-first — tests are TypeScript/Python/Groovy scripts (Playwright, Selenium)
Managed — a service writes the tests for you (QA Wolf)

Low-code sits between. You get readability and accessibility for non-engineers, plus optional code hooks when your team needs them.

Quick Comparison: Low-Code Test Automation Tools in 2026

Tool	Authoring Format	Self-Healing	AI Coding Agent Support	Best For
Shiplight AI	Intent-based YAML	Intent-based	Yes (MCP)	AI-native engineering teams
Mabl	Visual builder	Auto-healing	No	Product + QA teams in enterprise
Katalon	Record + optional scripts	Smart Wait	No	Mixed-skill teams needing breadth
testRigor	Plain English	NL re-interpretation	No	Non-technical QA teams
ACCELQ	Visual + NLP	AI-powered	No	Enterprises with heterogeneous stacks
Functionize	NLP + visual recording	ML-based	No	Large enterprises willing to train models
Virtuoso QA	Natural language	Autonomous AI	No	Teams needing visual + functional coverage

The 7 Best Low-Code Test Automation Tools in 2026

1. Shiplight AI — Low-Code for AI-Native Engineering Teams

Best for: Engineering teams building with AI coding agents who want low-code authoring with git-native storage.

Shiplight's authoring is genuinely low-code: tests are structured YAML with natural-language intent steps, readable by anyone who can follow a bulleted list. Optional CODE: blocks let engineers embed custom assertions when needed. The Shiplight Plugin exposes test generation and execution as Model Context Protocol (MCP) tools that Claude Code, Cursor, Codex, and GitHub Copilot can call directly.

goal: Verify user can complete checkout
steps:
  - intent: Log in as a test user
  - intent: Add the first product to the cart
  - intent: Proceed to checkout
  - intent: Complete payment with test card
  - VERIFY: order confirmation page shows order number

Strengths:

Intent-based self-healing — tests survive UI redesigns, not just minor locator changes
MCP integration — only low-code tool callable by AI coding agents
Tests live in your git repo — reviewable in PRs, portable, no vendor lock-in
Built on Playwright for real browser execution
SOC 2 Type II certified

Tradeoffs: Web only (no mobile device cloud). Newer platform than legacy low-code tools.

See Shiplight vs Mabl for a direct head-to-head on low-code alternatives.

2. Mabl — Visual Low-Code for Product + QA Teams

Best for: Enterprise product and QA teams wanting polished drag-and-drop authoring with built-in analytics.

Mabl is the most established visual low-code test automation platform. Its drag-and-drop builder generates tests from user stories and autonomous app exploration. Auto-healing, visual regression, and strong Jira integration round out a complete enterprise feature set.

Strengths: Clean visual authoring accessible to non-engineers. Built-in visual regression and accessibility testing. Strong Jira, GitHub, and GitLab integrations.

Tradeoffs: Tests live in Mabl's platform — not your git repo. No MCP integration. Cost scales with test volume.

For alternatives see Mabl alternatives.

3. Katalon — Flexible Low-Code with Optional Scripting

Best for: Large QA teams with mixed technical skills needing web, mobile, API, and desktop coverage from one platform.

Katalon is a long-standing low-code test automation platform. Its record-and-playback authoring handles simple cases without code; its Groovy/Java scripting support handles complex scenarios engineers want to customize. Smart Wait and AI-assisted locator generation reduce flakiness.

Strengths: Broad platform coverage, mature ecosystem, flexible authoring across skill levels, free tier available.

Tradeoffs: AI features are augmentation rather than generation — authoring is still largely manual. No MCP integration. Feel is more traditional than AI-native.

See Shiplight vs Katalon for a head-to-head.

4. testRigor — Plain-English Low-Code

Best for: Non-technical QA teams or business analysts who own testing without engineering support.

testRigor stretches the definition of low-code toward no-code — tests are plain-English sentences that the AI interprets at runtime. Covers web, mobile native, and API from one platform.

Strengths: Lowest barrier to entry — anyone who can write English can author tests. Broad platform coverage (web, mobile, API).

Tradeoffs: Plain-English ambiguity can produce unpredictable behavior on complex flows. Tests live in testRigor's platform. No MCP integration.

See Shiplight vs testRigor for a head-to-head.

5. ACCELQ — Codeless Cross-Platform Low-Code

Best for: Enterprises with heterogeneous stacks spanning web, mobile, API, SAP, and desktop.

ACCELQ's low-code authoring is codeless across the widest platform coverage on this list — including SAP and legacy desktop applications. Model-based test design and AI-powered self-healing work across all supported platforms.

Strengths: Broadest platform coverage. Codeless authoring accessible to non-engineers. Strong for SAP and legacy stacks.

Tradeoffs: Enterprise pricing. No MCP integration. Tests live in ACCELQ's platform.

See ACCELQ alternatives.

6. Functionize — ML-Driven Low-Code

Best for: Enterprises with complex applications willing to invest in application-specific ML training.

Functionize's low-code authoring uses NLP and visual recording. Its distinctive capability is ML training on your specific application — healing accuracy and test-generation quality improve the longer the system runs on your app.

Strengths: Application-specific ML accuracy improves over time. Strong enterprise features — SSO, RBAC, audit logs.

Tradeoffs: Training period before the model pays off. Enterprise-only pricing. Opaque ML decisions. No MCP integration.

See Functionize alternatives.

7. Virtuoso QA — Natural-Language Low-Code with Visual Testing

Best for: Teams that need autonomous low-code testing combined with a strong visual regression layer.

Virtuoso combines natural-language test authoring with autonomous visual testing. Its AI generates test steps from intent descriptions and continuously monitors for visual regressions without separate screenshot-comparison tooling.

Strengths: Natural language + visual testing in one platform. Autonomous test generation from user stories. Self-maintaining tests with change detection.

Tradeoffs: Tests live in Virtuoso's platform. No MCP integration. Enterprise-only pricing.

How to Choose a Low-Code Test Automation Tool

By team profile

Team profile	Best low-code fit
Engineers using AI coding agents	Shiplight AI
Product + QA teams wanting polished visual authoring	Mabl
Mixed-skill QA team needing broad coverage	Katalon
Non-technical QA / business analysts	testRigor
Enterprise with SAP / mobile / desktop	ACCELQ
Large enterprise willing to train ML models	Functionize
Teams where visual regression is business-critical	Virtuoso QA

By what "low-code" means to you

If you want…	Best fit
Tests-as-code in your git repo but low-code readable	Shiplight AI
Drag-and-drop visual authoring	Mabl
Record-and-playback with optional code extensions	Katalon
Plain-English sentences only	testRigor
Codeless for non-web applications	ACCELQ
ML-driven authoring with minimal human input	Functionize

By AI coding agent integration

Only Shiplight has native MCP integration today. If your team has adopted Claude Code, Cursor, Codex, or GitHub Copilot and wants low-code testing callable from the coding agent during development, Shiplight is the only option on this list that fits. Every other tool treats testing as a separate workflow from coding.

Low-Code vs No-Code vs Code-First Test Automation

A common confusion: "low-code" and "no-code" are not synonyms.

Approach	Definition	Example tools
No-code	Zero code at any stage	testRigor plain English, pure visual builders
Low-code	Primarily structured non-code with optional code extensions	Shiplight YAML, Mabl visual, Katalon record+scripts
Code-first	Tests are source code in a programming language	Playwright, Selenium, Cypress

Low-code is the most adopted category in 2026 because it balances accessibility (non-engineers contribute) with rigor (structured formats are deterministic). See what is no-code test automation? for the no-code side, and test authoring methods compared for all five authoring approaches side-by-side.

FAQ

What is low-code test automation?

Low-code test automation is a category of testing platforms where tests are authored primarily through structured non-code formats — visual builders, YAML with natural-language intent, or NLP sentences — with optional code extensions for complex scenarios. It sits between no-code (zero code) and code-first (Playwright/Selenium scripts), and is the most adopted authoring category in 2026 because it balances accessibility with rigor.

What is the difference between low-code and no-code test automation?

No-code test automation means zero coding at any stage — tests are pure plain English or visual recordings. Low-code means most authoring is non-code, but there are optional code extensions when complex logic is needed. testRigor is closer to no-code; Katalon and Shiplight are low-code because they support code extensions.

Which low-code test automation tool is best for AI coding agents?

Shiplight AI is the only low-code tool with native MCP integration. Its plugin exposes test generation and browser automation as MCP tools that Claude Code, Cursor, Codex, and GitHub Copilot can call during development. Other low-code tools treat testing as a separate workflow from coding. See best AI QA tools for coding agents for a deeper comparison.

Is low-code test automation reliable for production?

Yes. Mabl, Katalon, testRigor, Functionize, and ACCELQ have been in production at enterprise scale for years. Shiplight is newer but production-ready with SOC 2 Type II certification. The right question is not whether low-code works, but which tool matches your workflow and maturity needs.

Can non-engineers use low-code test automation tools?

Yes — that's the primary value proposition. Product managers, designers, QA analysts, and business users can author and review tests without writing code. See no-code testing for non-technical teams for a practical guide, which applies to low-code approaches as well.

How does low-code test automation handle complex flows like authentication or payments?

Most low-code tools handle authentication including OAuth, SSO, and 2FA out of the box. For truly complex scenarios (API-level setup before a UI flow, conditional logic based on runtime state), code extensions in low-code tools (Shiplight CODE: blocks, Katalon Groovy scripts) handle what visual authoring cannot. This is the key advantage of low-code over pure no-code.

Conclusion

Low-code test automation is the dominant authoring category in 2026 because it lets engineers and non-engineers contribute to the same test suite. The right tool depends on your team's workflow, platform coverage needs, and whether you're building with AI coding agents.

For teams building with AI coding agents, Shiplight AI is the clear first choice — it is the only low-code tool with native MCP integration, and its intent-based YAML format combines readability for non-engineers with the structure coding agents can generate. For teams with different priorities, Mabl, Katalon, testRigor, ACCELQ, Functionize, and Virtuoso QA each win for specific use cases.

Run a 30-day pilot on your highest-value user flow with two or three tools. Measure authoring time, healing success rate on UI changes, and maintenance burden — the numbers tell you which low-code test automation tool fits your team.

Get started with Shiplight Plugin.

Test Authoring Methods Compared: 5 Ways Automated Tests Are Written in 2026

Shiplight — Mon, 20 Apr 2026 21:13:59 +0000

Originally published on the Shiplight blog.

Test authoring is how automated tests get created — the process of translating what a product should do into executable checks that run in CI. In 2026, five methods coexist, each with distinct tradeoffs in speed, readability, maintenance, and who on the team can participate.

A test framework like Playwright or Selenium is only half the story. The other half is authoring — how you get the tests into existence in the first place. In 2026, five authoring methods dominate:

Code-first (Playwright, Selenium, Cypress scripts)
Record-and-playback
Plain English / NLP test steps
AI-generated tests from specs or UI exploration
Intent-based YAML

None of these is universally best. The right method depends on who writes the tests, how often the product changes, and whether AI coding agents are part of your development workflow. This guide covers all five with concrete examples and a decision framework.

Method 1: Code-First Test Authoring

Code-first authoring means engineers write tests directly in a programming language — TypeScript, JavaScript, Python, Groovy — using a test framework's API to interact with the browser.

This is the original model. Playwright, Selenium, Cypress, and WebDriver all target this approach.

import { test, expect } from '@playwright/test';

test('user can complete checkout', async ({ page }) => {
  await page.goto('https://app.example.com');
  await page.getByLabel('Email').fill('test@example.com');
  await page.getByLabel('Password').fill('password123');
  await page.getByRole('button', { name: 'Sign in' }).click();
  await page.getByRole('link', { name: 'Add to cart' }).click();
  await page.getByRole('button', { name: 'Checkout' }).click();
  await expect(page.getByText('Order confirmed')).toBeVisible();
});

Strengths: Maximum control over browser behavior, deterministic execution, full access to framework features, works well in CI.

Weaknesses: Engineers-only — product managers, designers, and QA analysts without coding skills cannot contribute. Tests break frequently when locators change, creating high maintenance cost. Authoring a new test from scratch takes hours.

Best for: Engineering-heavy teams with dedicated test infrastructure and the headcount to maintain it.

Method 2: Record-and-Playback Test Authoring

Record-and-playback test authoring means the tool observes your manual browser interactions and generates a runnable test script from them. You click through the flow, the tool captures each action, and the output is an executable test.

This approach is ~20 years old — Selenium IDE pioneered it, and most modern no-code tools (Katalon, some modes of ACCELQ) still use variants of it. AI-augmented record-and-playback adds smart locator generation and auto-healing.

Typical flow:

Click "Record" in the tool
Perform the test manually — log in, click buttons, fill forms
Tool generates a test with steps mirroring your actions
Replay to verify

Strengths: Fast initial authoring. Non-engineers can produce test drafts. No coding required.

Weaknesses: Generated tests are often brittle — recorded click coordinates or CSS selectors break when the UI changes. Tests drift from user intent because what was recorded was a specific execution, not a specification of behavior. Difficult to maintain at scale.

Best for: Quick initial coverage, documenting existing workflows, or onboarding non-engineers into test creation.

Codeless E2E testing covers how modern record-and-playback has evolved.

Method 3: Plain English / NLP Test Authoring

Plain English test authoring means writing tests as natural-language sentences that the tool interprets and translates into browser actions at runtime.

No code, no YAML, no selectors. Just prose.

Go to https://app.example.com/login
Enter "admin@example.com" into "Email"
Enter "password123" into "Password"
Click "Sign In"
Check that the page contains "Welcome, Admin"

testRigor pioneered this model. Some features of Virtuoso QA, Functionize, and ACCELQ offer similar authoring experiences.

Strengths: Anyone who can write a bulleted list can create a test. Highest accessibility for non-technical team members — business analysts, product managers, support staff. Tests read like documentation.

Weaknesses: Ambiguity — "Click Sign In" assumes the tool can resolve which element is "Sign In" when there might be multiple. Complex flows with dynamic content, custom components, or non-standard UI patterns challenge natural-language resolution. Debugging unclear tests is harder than debugging code.

Best for: Non-technical QA teams, business-rule-driven testing, environments where tests need to be readable by non-engineers.

See no-code testing for non-technical teams for a deeper guide.

Method 4: AI-Generated Tests from Specs or UI Exploration

AI-generated test authoring means the AI produces test cases automatically from inputs like product specifications, user stories, or autonomous application exploration — with no manual step-by-step authoring.

Three input types are common:

From specifications

You feed the AI a user story, acceptance criteria, or PRD section. It generates a test covering the described behavior.

User story: "As a signed-in user, I can add items to my cart and complete checkout with a saved payment method."

→ AI produces a 10-step test covering login, navigation, add-to-cart, checkout form, payment confirmation.

From UI exploration

The AI navigates your running application, discovers flows, and generates tests for what it finds. Mabl and some Functionize modes work this way. No input required beyond a URL.

From session recordings

The AI observes real user traffic and generates tests reflecting actual usage patterns. Checksum is the primary example.

Strengths: Scales — coverage grows without human authoring effort. Captures flows that engineers wouldn't think to write tests for. Integrates naturally with AI coding agent workflows.

Weaknesses: Generated tests may include redundant or low-value cases. Spec-to-test accuracy depends on spec clarity. Autonomous exploration can miss business-critical edge cases that aren't obvious from the UI.

Best for: Teams with limited QA headcount, SaaS products with established user bases, or engineering organizations that want coverage to scale with development velocity.

See AI testing tools that automatically generate test cases for a tool-by-tool comparison.

Method 5: Intent-Based YAML Test Authoring

Intent-based YAML test authoring means writing tests as structured YAML files where each step describes user intent in natural language, with AI resolving intent to browser actions at runtime.

This is the approach Shiplight is built around. It combines the readability of plain English with the structure and version-control friendliness of code.

goal: Verify user can complete checkout
steps:
  - intent: Log in as a test user
  - intent: Navigate to the product catalog
  - intent: Add the first product to the cart
  - intent: Proceed to checkout
  - intent: Enter shipping address
  - intent: Complete payment with test card
  - VERIFY: order confirmation page shows order number

Tests are readable by anyone who can follow a bulleted list, yet structured enough to live in git, appear in pull request diffs, and run in CI. When the UI changes, Shiplight resolves each intent step from scratch rather than failing on a stale selector — the intent-cache-heal pattern.

Intent-based YAML is the primary authoring model in Shiplight Plugin, which exposes /create_e2e_tests as an MCP tool so Claude Code, Cursor, Codex, and GitHub Copilot can generate intent-based tests during development.

Strengths: Readable like plain English, structured like code. Survives UI changes via intent-based self-healing. Version-controlled, reviewable in PRs, portable across environments. Can be generated by AI coding agents or written by non-engineers.

Weaknesses: Requires basic YAML familiarity (less than a scripting language, more than plain prose). Newer format with smaller ecosystem than Playwright or Selenium scripts.

Best for: Teams using AI coding agents, mixed-skill engineering organizations, and any team that wants tests as a first-class artifact in their git workflow.

Test Authoring Methods: Side-by-Side Comparison

Method	Who Authors	Format	Readability	Maintenance	AI Agent Support
Code-first	Engineers	Code (TS/JS/Python)	Low (non-engineers)	Manual	Limited
Record-and-playback	Anyone	Recorded script	Medium	Fragile	No
Plain English / NLP	Anyone	Natural language	High	Self-healing typical	Limited
AI-generated	AI	Varies (code or proprietary)	Varies	Self-healing typical	Partial
Intent-based YAML	Anyone or AI	YAML with intent steps	High	Intent-based self-healing	Native (MCP)

How to Choose a Test Authoring Method

By team profile

Team profile	Recommended method
All engineers, need max control	Code-first (Playwright)
QA team with no coding	Plain English / NLP or intent-based YAML
Engineers + AI coding agents	Intent-based YAML (Shiplight)
Want coverage without authoring	AI-generated (exploration or session-based)
Need to onboard non-engineers gradually	Record-and-playback, graduate to YAML

By application change velocity

Stable UI, rare changes: Code-first or record-and-playback both work
High change velocity: Self-healing methods (plain English, intent-based YAML, AI-generated)
AI coding agents driving changes: Intent-based YAML with MCP integration

By review requirements

Tests reviewed by product managers: Plain English or intent-based YAML
Tests reviewed by engineers only: Any method works
Regulated industries (audit trail required): Intent-based YAML (git-native, version-controlled, human-readable)

FAQ

What is test authoring?

Test authoring is the process of creating automated tests — translating what a product should do into executable checks that run in a test framework. It is distinct from test execution (which runs the tests) and test maintenance (which fixes them when they break).

Is record-and-playback still used in 2026?

Yes, but it has evolved. Modern AI-augmented record-and-playback tools add smart locator generation and self-healing to reduce the brittleness that made the original approach unreliable. It remains useful for quick initial coverage and onboarding non-engineers, but has been displaced for production suites by intent-based and AI-generated methods.

What is the difference between plain English test authoring and intent-based YAML?

Plain English tests are unstructured prose — the tool parses each sentence and infers actions. Intent-based YAML is structured: each step is a YAML key-value pair with a clear intent field, making it version-control-friendly and unambiguous to parse. Intent-based YAML is a middle ground between the flexibility of plain English and the rigor of code.

Can AI coding agents generate tests directly?

Yes, with the right authoring format and integration. Shiplight Plugin exposes test generation as an MCP tool that Claude Code, Cursor, Codex, and GitHub Copilot can call during development — the coding agent generates intent-based YAML tests as part of the same task it uses to implement a feature.

Should I use multiple authoring methods in one project?

It's common. Many teams use code-first Playwright tests for infrastructure-level flows, intent-based YAML for UI-level E2E, and AI-generated tests for coverage breadth. The key is consistency within each category — don't mix authoring methods for the same type of test.

Conclusion

The choice of test authoring method is a higher-leverage decision than most teams realize. It determines who on the team can contribute, how often tests break, and whether your test suite scales with development velocity or against it.

For teams building with AI coding agents, intent-based YAML is the strongest fit — it combines the readability non-engineers need with the structure AI agents can generate, and the self-healing that makes tests survive high-velocity UI changes.

Try intent-based YAML testing with Shiplight Plugin — installs into Claude Code, Cursor, Codex, and GitHub Copilot in a few minutes.

Agent-Native Autonomous QA: The New Paradigm for Software Quality in 2026

Shiplight — Sun, 19 Apr 2026 21:01:40 +0000

Originally published on the Shiplight blog.

Two terms describe where software quality assurance is heading in 2026: agent-native and autonomous QA. They describe the same shift from different angles. Agent-native is about architecture — QA tools that AI coding agents can invoke directly, rather than dashboards humans operate. Autonomous QA is about operation — a quality system that runs, heals, and maintains itself without a human in the loop for each step.

Together they define a new category: agent-native autonomous QA. This is the model QA must adopt to keep up with teams building software using AI coding agents like Claude Code, Cursor, Codex, and GitHub Copilot.

This guide explains what each term means, why they matter together, and what a production-ready agent-native autonomous QA system looks like.

What "Agent-Native" Means

Agent-native describes software tools designed so AI agents can use them as peers — invoking capabilities, interpreting output, and incorporating results into an ongoing task — through agent-callable interfaces rather than human dashboards. Agent-native QA tools expose their functionality via Model Context Protocol (MCP) or equivalent protocols.

Contrast with two older models:

Human-native tools are built for people. A QA engineer logs into a dashboard, configures a test run, reviews a report. The tool has no API surface an AI agent can use meaningfully.

AI-augmented tools use AI internally to help humans — smart locators, test suggestions, auto-complete for test scripts. The AI lives inside the tool but doesn't expose the tool to external agents.

Agent-native tools are built so AI agents are first-class users. The Shiplight Plugin is agent-native: its browser automation, test generation, and review capabilities are exposed as MCP tools that Claude Code, Cursor, Codex, and GitHub Copilot can call directly during development.

Agent-native QA in practice

When the coding agent is building a feature, it can:

Call /verify — Shiplight opens a real browser and confirms the UI change looks and behaves correctly
Call /create_e2e_tests — Shiplight generates a self-healing test covering the new flow
Call /review — Shiplight runs automated reviews across security, accessibility, and performance

The agent chains these together as part of its development task. No human context switch. No separate QA phase. No dashboard.

What "Autonomous QA" Means

Autonomous QA is software quality assurance where AI agents handle the entire testing loop — deciding what to test, generating tests, executing them, interpreting results, and healing broken tests — without human intervention at each step. The human role is oversight, not execution.

In practice, an autonomous QA system:

Decides what to test — based on code changes, specifications, or observed behavior
Generates tests — from natural language intent, not manual scripting
Executes tests — in a real browser, against the actual application
Interprets results — distinguishes genuine failures from flakiness
Heals broken tests — when the UI changes, resolves the correct element from stored intent rather than failing on a stale selector

The human role shifts from execution to oversight: reviewing the system's output, making go/no-go calls, setting quality policies. Everything in between is handled by the agent.

This is different from AI-assisted QA, where humans still drive each step and AI only accelerates parts of the workflow. In autonomous QA, the AI is the driver.

Why Agent-Native and Autonomous QA Matter Together

Either one alone is insufficient.

Autonomous QA without agent-native tooling still works, but it operates as a separate system from development. The coding agent builds, then a QA system runs later in CI. Feedback is delayed. Coverage gaps happen because the QA system doesn't know what the coding agent just changed.

Agent-native tooling without autonomy means the coding agent can call the QA tool, but humans still need to write, maintain, and triage the tests. The agent's calls just trigger more work for humans downstream.

Combining them produces the pattern that matters for agent-first development:

Coding agent writes code
Coding agent calls agent-native QA tool to verify
QA tool autonomously generates coverage, runs tests, interprets results, heals broken tests
Coding agent incorporates QA results into its task
Human reviews the completed PR — code and tests together

The human is present at exactly one step: final review. Everything else — implementation and verification — is handled autonomously by agents using agent-native tools.

Traditional QA vs. AI-Assisted QA vs. Agent-Native Autonomous QA

Capability	Traditional QA	AI-Assisted QA	Agent-Native Autonomous QA
Test authoring	Engineer writes code	AI suggests, human writes	AI generates from intent
Test maintenance	Manual locator fixes	AI-suggested fixes	Autonomous intent-based healing
Triggered by	Human in CI	Human in CI	Coding agent during development
Interface	Human dashboard	Human dashboard	MCP tools for agents
Human role	Drives every step	Drives steps, AI assists	Reviews output, sets policy
Feedback loop	Hours to days	Hours	Minutes — inside dev loop
Scales with dev velocity	No	Partially	Yes

What an Agent-Native Autonomous QA System Looks Like

Concrete components of a production system:

1. An agent-callable interface

The QA system exposes its capabilities as MCP tools, APIs, or equivalent. AI coding agents can call those tools as part of their autonomous task execution. Human dashboards are optional, not primary.

2. Intent-based test authoring

Tests describe what should happen, not how to click. Intent is portable across UI changes. A test that says intent: Click the Save button survives when the button's CSS class changes, because the agent re-resolves the element from intent at runtime.

Example from Shiplight's YAML test format:

goal: Verify user can complete onboarding
steps:
  - intent: Navigate to the signup page
  - intent: Fill in name, email, and password
  - intent: Submit the registration form
  - intent: Complete the product tour steps
  - VERIFY: user lands on the dashboard with their name shown

3. Real browser execution

Built on Playwright or equivalent for reliability. Tests run against the actual application, not synthetic environments. Screenshots, traces, and step-by-step execution logs are available when failures occur.

4. Intent-based self-healing

When a locator fails, the system re-resolves the correct element from stored intent using AI. Self-healing based on intent handles UI redesigns, not just minor locator changes. Locator-fallback healing (most legacy tools) only handles small variations.

5. Git-native test artifacts

Tests live in your repository, appear in pull request diffs, and are reviewable by non-engineers. Tests in proprietary vendor databases can't be reviewed in code review and create lock-in.

6. CI/CD integration via CLI

The system runs in any CI environment — GitHub Actions, GitLab CI, CircleCI, Jenkins — via CLI. No vendor-locked runners required.

Who Needs Agent-Native Autonomous QA?

Teams where:

AI coding agents are generating code faster than QA can verify it. Without agent-native QA, coverage gaps grow. With it, the coding agent verifies its own work.

Test maintenance is consuming engineering time. Teams typically spend 40–60% of QA effort fixing tests broken by routine UI changes. Autonomous intent-based healing eliminates this category of work.

Release cadence is blocked by manual QA handoffs. Autonomous QA embedded in the development loop removes the QA cycle from the critical path.

Enterprise teams need compliance plus velocity. Agent-native autonomous QA with SOC 2 Type II certification, RBAC, SSO, and audit logs lets enterprises ship at startup speed without compliance compromise. See our enterprise self-healing test automation guide for how this works in regulated environments.

FAQ

What is agent-native QA?

Agent-native QA is quality assurance tooling designed so AI coding agents can invoke it directly as part of their autonomous task execution. It exposes capabilities through MCP or equivalent agent-callable interfaces rather than human-only dashboards. Shiplight Plugin is an example: its /verify, /create_e2e_tests, and /review commands can be called by Claude Code, Cursor, Codex, or GitHub Copilot during development.

What is autonomous QA?

Autonomous QA is a model where AI handles the full quality assurance loop — deciding what to test, generating tests, executing them, interpreting results, and healing broken tests — without human intervention at each step. Humans provide oversight and judgment, not execution. See agentic QA testing for the full definition and how it differs from AI-assisted testing.

How is agent-native different from AI-powered testing tools?

AI-powered tools use AI internally (smart locators, test suggestions, auto-complete) but are operated by humans through dashboards. Agent-native tools expose their capabilities so AI agents can use them as peers — the AI is an external user, not an internal feature. This distinction matters because agent-first development workflows need QA tools that coding agents can call directly.

Can I get agent-native autonomous QA with existing tools like Playwright or Selenium?

Partially. Playwright and Selenium are excellent execution engines, but they are not autonomous — they run tests humans wrote. To get agent-native autonomous QA you need a layer above them that handles test generation, intent-based healing, and exposes agent-callable interfaces. Shiplight is built on Playwright and adds those layers.

Is agent-native autonomous QA production-ready?

Yes. Teams using Shiplight Plugin with AI coding agents are shipping production software today. SOC 2 Type II certification, enterprise SSO, RBAC, and audit logs are available for regulated industries. See enterprise-grade agentic QA for the full enterprise readiness framework.

Conclusion

Agent-native and autonomous QA are not two separate capabilities — they are two requirements for the same new category of tooling. QA that is agent-native but not autonomous still creates work for humans downstream. QA that is autonomous but not agent-native cannot participate in the agent-first development loop.

Teams building with AI coding agents need both. Shiplight is purpose-built for this: agent-native via MCP integration, autonomous via intent-based generation and self-healing, and production-ready with SOC 2 Type II certification.

Get started with agent-native autonomous QA

How to Evaluate AI Test Generation Tools: A Buyer's Guide

Shiplight — Wed, 15 Apr 2026 00:18:45 +0000

Evaluating AI test generation tools — running a structured eval against real criteria rather than vendor demos — is the only way to know which tool will hold up in production. The AI industry has converged on structured evals as the standard for assessing AI system quality, whether for LLMs or for the agents that use them. The same discipline applies to test generation tools: Anthropic's guide to demystifying evals for AI agents and OpenAI's evaluation best practices both emphasize measuring real-world output quality over capability claims. The same principle applies when you are choosing a test generation platform.

Why Evaluation Matters More Than Ever

Dozens of AI test generation tools now promise to generate end-to-end tests automatically. The claims are similar. The underlying approaches are not.
Choosing the wrong tool creates compounding costs: vendor lock-in, test suites needing constant maintenance, or generated tests that miss critical business logic. This guide provides a seven-dimension eval checklist based on the criteria that matter in production, not in demos.

The Seven-Dimension Evaluation Framework

1. Test Quality

The most important and most overlooked question: are the generated tests actually good?
What to evaluate:

Assertion depth -- Does the tool verify text content, state changes, and data integrity, or just "element is visible"?
Flow completeness -- Does it cover setup, action, and teardown, or produce fragments requiring assembly?
Determinism -- Do the same inputs produce the same tests?
Readability -- Can an engineer understand the generated test without consulting documentation? Red flag: Tools that demo well on simple forms but produce shallow tests on complex workflows. Ask for tests against your own application. See our guide on what AI test generation involves. ### 2. Maintenance Burden Generating tests is easy. Keeping them working as your application evolves is the real challenge. What to evaluate:
Self-healing capability -- Does it repair tests automatically? Simple locator fallbacks or intent-based resolution?
Update workflow -- Can you regenerate selectively, or must you regenerate the entire suite?
Version control integration -- Are tests stored as committable, diffable files?
Change visibility -- Can you see what was healed and why? Red flag: Tools that heal silently without an audit trail. ### 3. CI/CD Integration What to evaluate:
Pipeline compatibility -- CLI, Docker, GitHub Action? Works with any CI system?
Parallelization -- Can tests run across multiple workers?
Reporting -- Standard output formats (JUnit XML, JSON) for existing dashboards?
Gating -- Can test results gate deployments with configurable thresholds? Red flag: Proprietary or cloud-only execution environments that prevent local debugging. ### 4. Pricing Model What to evaluate:
Per-seat vs. per-test vs. per-execution -- Per-test pricing penalizes coverage; per-execution penalizes frequent testing
Included AI credits -- Understand what incurs overage charges
Tier boundaries -- Are self-healing, CI/CD, or SSO gated behind enterprise tiers?
Total cost of ownership -- Include training, migration, and ongoing operational costs Red flag: Opaque pricing requiring a sales call. Essential features locked behind enterprise contracts. ### 5. Vendor Lock-In What to evaluate:
Test portability -- Standard Playwright tests, or proprietary format?
Data ownership -- Can you export test definitions and execution history?
Framework dependency -- Standard frameworks or proprietary runtime?
Migration path -- Do tests survive if you stop using the tool? Red flag: Proprietary formats with no export. No documented migration path. Shiplight addresses lock-in by generating standard Playwright tests and operating as a plugin layer rather than a replacement platform. ### 6. Self-Healing Capability What to evaluate:
Healing approach -- Locator fallbacks, AI-driven resolution, or intent-based healing?
Healing coverage -- What percentage of failures does it heal? Ask for production metrics, not lab results
Healing transparency -- Can you see what changed and approve it?
Healing speed -- Inline during execution, or a separate post-failure step? For a deep comparison, see our AI-native E2E buyer's guide. ### 7. AI Coding Agent Support What to evaluate:
Agent-triggered testing -- Can AI coding agents trigger test generation or execution automatically?
PR integration -- Are AI-generated code changes validated automatically in pull requests?
Feedback loop -- Can test results feed back to the coding agent to fix issues it introduced?
API accessibility -- Does the tool expose APIs agents can invoke programmatically? Red flag: Tools designed only for human-driven workflows with no programmatic interface. See our guide on the best AI testing tools in 2026 for tools that score well on agent support. ## The Evaluation Scorecard Use this scorecard to rate each tool on a 1-5 scale across all seven dimensions: | Dimension | Weight | Tool A | Tool B | Tool C | |---|---|---|---|---| | Test Quality | 25% | _/5 | _/5 | _/5 | | Maintenance Burden | 20% | _/5 | _/5 | _/5 | | CI/CD Integration | 15% | _/5 | _/5 | _/5 | | Pricing Model | 10% | _/5 | _/5 | _/5 | | Vendor Lock-In | 15% | _/5 | _/5 | _/5 | | Self-Healing | 10% | _/5 | _/5 | _/5 | | AI Agent Support | 5% | _/5 | _/5 | _/5 | | Weighted Total | 100% | | | | Weight each dimension according to your team's priorities. Teams with large existing test suites should weight maintenance burden higher. Teams in regulated industries should weight test quality and vendor lock-in higher. ## Key Takeaways
Test quality is the most important dimension -- a tool that generates shallow tests provides false confidence
Self-healing sophistication varies dramatically -- intent-based healing covers far more scenarios than locator fallbacks
Vendor lock-in is the hidden cost -- prioritize tools that generate portable, standard test code
CI/CD integration must be seamless -- friction in the pipeline kills adoption
AI coding agent support is increasingly essential -- choose tools that work programmatically, not just through UIs
Evaluate against your own application -- demo environments are designed to make every tool look good ## Frequently Asked Questions ### How many tools should I evaluate? Evaluate three in depth. Start with a longlist of 5-6, narrow based on documentation and pricing, then run hands-on evaluations with your actual application. ### Should I run a paid pilot or rely on free trials? Always pilot against your actual application. A two-week pilot with 20-30 tests against your real UI is worth more than months of feature comparison spreadsheets. ### How long should the evaluation take? Four to six weeks: one week for research, one week to narrow to three finalists, and two to three weeks for hands-on evaluation. ### What is the biggest evaluation mistake? Optimizing for test creation speed instead of maintenance cost. A tool that generates 100 tests in 10 minutes but requires 20 hours per week of maintenance is worse than one that generates in an hour but maintains itself. Evaluate 12-month total cost of ownership. ## Get Started Ready to evaluate Shiplight against your current testing stack? Request a demo with your own application and see how the seven-dimension framework applies to your specific situation. Explore the Shiplight plugin ecosystem and see how AI test generation works in practice with standard Playwright tests. For a side-by-side comparison of tools that auto-generate test cases, see AI testing tools that automatically generate test cases.

References: Playwright Documentation · Anthropic: Demystifying Evals for AI Agents · OpenAI: Evaluation Best Practices

Deterministic E2E Testing in an AI World: The Intent, Cache, Heal Pattern

Shiplight — Tue, 14 Apr 2026 16:56:57 +0000

End-to-end tests are supposed to be your final confidence check. In practice, they often become a recurring tax: brittle selectors, flaky timing, and one more dashboard nobody trusts.
AI has promised a reset. But most teams have a reasonable concern: if a model is “deciding” what to click, how do you keep results deterministic enough to gate merges and releases?
The answer is not choosing between rigid scripts and free-form AI. It is designing a system where intent is the source of truth, deterministic replay is the default, and AI is the safety net when reality changes.
This is the core idea behind Shiplight AI’s approach to agentic QA: stable execution built on intent-based steps, locator caching, and self-healing behavior that keeps tests working as your UI evolves.
Below is a practical model you can apply immediately, plus how Shiplight supports each layer across local development, cloud execution, and AI coding agent workflows.

Why E2E Tests Break: Two Distinct Failure Modes

When an end-to-end test fails, teams usually treat it like a single category: “the test is red.” In reality, there are two fundamentally different failure modes:

The product is broken. The user journey no longer works.
The test is broken. The journey still works, but the automation got lost due to UI drift, timing, or stale locators. Classic UI automation makes these two failure modes hard to separate because the test definition is tightly coupled to implementation details. If the DOM changes, the test fails the same way it would if checkout genuinely broke. Shiplight’s design goal is to decouple those concerns by writing tests around what a user is trying to do, then treating selectors as an optimization, not the test itself. ## The pattern: Intent, Cache, Heal ### 1) Intent: write what the user does, not how the DOM is structured Shiplight tests can be authored in YAML using natural language statements. At the simplest level, a test defines a goal, a starting URL, and a list of steps, including VERIFY: assertions. A simplified example looks like this:

goal: Verify user journey
statements:
 - intent: Navigate to the application
 - intent: Perform the user action
 - VERIFY: the expected result

This intent-first layer is readable enough for engineers, QA, and product to review together, which is where quality should start. For more on making tests reviewable in pull requests, see The PR-Ready E2E Test.

2) Cache: replay deterministically when nothing has changed

Pure natural language execution is powerful, but you do not want your CI pipeline to “reason” about every click on every run.
Shiplight addresses this with an enriched representation where steps can include cached Playwright-style locators inside action entities. The key concept from Shiplight’s docs is worth adopting as a general rule:
Locators are a cache, not a hard dependency. (For a deeper exploration of this mental model, see Locators Are a Cache.)
When the cache is valid, execution is fast and deterministic. When it is stale, you still have intent to fall back on.
Shiplight also runs on top of Playwright, which gives teams a familiar, proven browser automation foundation. Teams looking for alternatives to raw Playwright scripting can explore Playwright Alternatives for No-Code Testing.

3) Heal: fall back to intent, then update the cache

UI changes are inevitable: a button label changes, a layout shifts, a component library gets upgraded.
Shiplight’s agentic layer can fall back to the natural language description to locate the right element when a cached locator fails. On Shiplight Cloud, once a self-heal succeeds, the platform can update the cached locator so future runs return to deterministic replay. For a deeper look at how this compares to other healing approaches, see What Is Self-Healing Test Automation.
This is how you stop paying the “daily babysitting” tax without sacrificing the reliability standards required for CI.

Making the pattern real: a practical rollout checklist

Here is a rollout approach that keeps scope controlled while compounding value quickly.

Step 1: Start with release-critical journeys, not “test coverage”

Pick 5 to 10 flows that create real business risk when broken: signup, login, checkout, upgrade, key settings changes. Write these as intent-first tests before you worry about breadth.

Step 2: Use variables and templates to avoid test suite sprawl

As soon as you have repetition, standardize it.
Shiplight supports variables for dynamic values and reuse across steps, including syntax designed for both generation-time substitution and runtime placeholders. It also supports Templates (previously called “Reusable Groups”) so teams can define common workflows once and reuse them across tests, with the option to keep linked steps in sync.
This is how you prevent your E2E suite from becoming 200 slightly different versions of “log in.”

Step 3: Debug where developers already work

Shiplight’s VS Code Extension lets you create, run, and debug *.test.yaml files with an interactive visual debugger directly inside VS Code, including step-through execution and inline editing.
This matters because reliability is not just about test execution. It is also about shortening the loop from “something failed” to “I understand why.”

Step 4: Integrate into CI with a real gating workflow

Shiplight provides a GitHub Actions integration built around API tokens, environment IDs, and suite IDs, so you can run tests on pull requests and treat results as a first-class CI signal.
Once the suite is stable, add policies like “block merge on critical suite failure” and “run full regression nightly.” Make quality visible and enforceable.

Step 5: Cut triage time with AI summaries

Shiplight Cloud includes an AI Test Summary feature that analyzes failed test results and provides root-cause guidance using steps, errors, and screenshots, with summaries cached after the first view for fast revisits.
This is not just convenience. It is how E2E becomes decision-ready instead of investigation-heavy.

Where Shiplight fits depending on how your team ships

Shiplight is designed to meet teams where they are:

Shiplight Plugin is built to work with AI coding agents, ingesting context (requirements, code changes, runtime signals), validating features in a real browser, and closing the loop by feeding diagnostics back to the agent.
Shiplight AI SDK extends existing Playwright-based test infrastructure rather than replacing it, emphasizing deterministic, code-rooted execution while adding AI-native stabilization and self-healing.
Shiplight Desktop (macOS) runs the Shiplight web UI while executing the browser sandbox and agent worker locally for fast debugging, and includes a bundled MCP server for IDE connectivity. ## The bottom line: AI should reduce uncertainty, not introduce it If your test system depends on brittle selectors, you will keep paying maintenance forever. If it depends on free-form AI decisions, you will struggle to trust results. The Intent, Cache, Heal pattern is the middle path that works in production: humans define intent, systems replay deterministically, and AI intervenes only when the app shifts underneath you. Shiplight AI is built around that philosophy, from YAML-based intent tests and locator caching to self-healing execution, CI integrations, and agent-native workflows. See how Shiplight compares to other AI testing approaches in Best AI Testing Tools in 2026. ## Intent, Cache, Heal: Key Takeaways
Verify in a real browser during development. Shiplight Plugin lets AI coding agents validate UI changes before code review.
Generate stable regression tests automatically. Verifications become YAML test files that self-heal when the UI changes.
Reduce maintenance with AI-driven self-healing. Cached locators keep execution fast; AI resolves only when the UI has changed.
Integrate E2E testing into CI/CD as a quality gate. Tests run on every PR, catching regressions before they reach staging. ## Frequently Asked Questions ### What is AI-native E2E testing? AI-native E2E testing uses AI agents to create, execute, and maintain browser tests automatically. Unlike traditional test automation that requires manual scripting, AI-native tools like Shiplight interpret natural language intent and self-heal when the UI changes. ### How do self-healing tests work? Self-healing tests use AI to adapt when UI elements change. Shiplight uses an intent-cache-heal pattern: cached locators provide deterministic speed, and AI resolution kicks in only when a cached locator fails — combining speed with resilience. ### What is MCP testing? MCP (Model Context Protocol) lets AI coding agents connect to external tools. Shiplight Plugin enables agents in Claude Code, Cursor, or Codex to open a real browser, verify UI changes, and generate tests during development. ### How do you test email and authentication flows end-to-end? Shiplight supports testing full user journeys including login flows and email-driven workflows. Tests can interact with real inboxes and authentication systems, verifying the complete path from UI to inbox. ## Get Started
Try Shiplight Plugin
Book a demo
YAML Test Format

Agentic QA Benchmark: How to Measure What Matters (2026)

Shiplight — Mon, 13 Apr 2026 02:12:26 +0000

Evaluating an agentic QA platform is harder than it looks. Every vendor can generate a test in a demo. What you cannot see in a demo is how that test performs three months later, after the agent has refactored the component four times and the test suite has grown to 200 cases. That is the real benchmark for agentic QA — not the first run, but the hundredth.

The right evaluation framework looks at five dimensions: heal rate, CI pass rate, coverage growth velocity, maintenance burden, and mean time to resolution on failures. Together, these metrics tell you whether a platform will compound value over time or accumulate hidden debt.

Why Standard QA Benchmarks Fail for Agentic Systems

Traditional QA benchmarks measure static properties: does the tool support your browsers? Can it integrate with your CI? Does it have a visual recorder? These matter, but they measure capability at a point in time, not performance over time.

Agentic QA platforms are fundamentally different because they operate in a feedback loop with a changing application. An agentic QA system generates tests, runs them, heals failures, and expands coverage — continuously. The benchmark question is not "what can it do?" but "what does it do to your test suite over 90 days?"

The five metrics below answer that question directly.

Benchmark Metric 1: Self-Heal Rate Under Real UI Change

Definition: The percentage of test failures caused by UI changes (not genuine regressions) that the platform resolves automatically without human intervention.

Why it matters: This is the primary maintenance cost driver. A platform with a 60% heal rate means 40% of UI-change-induced failures require manual intervention. At scale, that is a significant engineering tax. A platform with a 90%+ heal rate means your test suite survives most UI changes automatically.

How to benchmark it:

Run a structured proof-of-concept:

Record the current state of the application and your test suite
Make a series of UI changes of increasing severity: rename a CSS class → change a button label → restructure a component → redesign a section
Measure what percentage of test failures heal automatically at each severity level

The severity gradient matters. Rule-based healing (locator fallback) handles minor changes well. Intent-based healing — like Shiplight's intent-cache-heal pattern — handles major restructuring that breaks every recorded locator.

Reference benchmarks:

Minor DOM changes (label rename, class change): 90–99% heal rate across most tools
Component restructure (parent container changes): 60–90% varies significantly by approach
Full section redesign: <40% for rule-based tools, 70–85% for intent-based tools

Benchmark Metric 2: CI Pass Rate Stability Over 90 Days

Definition: The percentage of CI runs that complete without human intervention (no test disabling, no manual locator fixes, no skip lists growing) over a 90-day period.

Why it matters: A test suite that requires weekly manual maintenance is a liability, not an asset. The benchmark is whether your CI pass rate holds steady as the application evolves — not just on day one.

How to benchmark it:

If the vendor offers a trial or PoC environment, run your actual test suite against your actual application for 4–8 weeks. Track:

How many tests were disabled or skipped vs. the baseline
How many manual locator fixes were required
Whether the CI pass rate trended up, flat, or down over time

A platform that shows a downward trend in CI pass rate over 30 days is a maintenance burden by month three. A platform that holds steady or improves as the self-healing cache warms is a compounding asset.

Benchmark Metric 3: Coverage Growth Velocity

Definition: The rate at which new test coverage is added per week, measured in distinct user flows covered, without proportionally increasing maintenance burden.

Why it matters: The promise of agentic QA is that coverage scales with the application without scaling the engineering effort required to maintain it. This metric tests whether that promise holds in practice.

How to benchmark it:

Count the number of distinct user flows covered at the start of the trial and at the end. Divide by the engineering hours invested in writing, reviewing, and maintaining tests during that period. The ratio — flows covered per engineering hour — is your coverage growth velocity.

A high-velocity platform adds 5–10 new flows per week with minimal manual effort. A low-velocity platform requires significant human involvement to add each new test, limiting how far coverage can grow.

Platforms that store tests as YAML files in your repository typically outperform proprietary platforms here because tests can be generated by AI agents directly and reviewed in the same workflow as code changes.

Benchmark Metric 4: Maintenance Hours Per Week

Definition: The engineering time spent per week on test maintenance — fixing broken tests, updating selectors, investigating false positives, and managing skip lists.

Why it matters: This is the most direct measure of hidden cost. A platform that claims to eliminate maintenance but requires 10 hours/week of engineering time is not delivering on the promise.

How to benchmark it:

Before the PoC, measure your current maintenance burden — how many hours per week does your team spend on broken tests, locator updates, and skip list management? This is your baseline.

During the PoC, track the same metric. The benchmark is whether the agentic platform reduces your maintenance burden measurably. Industry data suggests teams spend 30–40% of testing effort on maintenance with traditional automation. An effective agentic QA platform should reduce this to under 10%.

Benchmark Metric 5: Mean Time to Resolution on Test Failures

Definition: The average time from "a test fails in CI" to "the failure is diagnosed and resolved" — either by healing automatically or by surfacing enough context for a developer or agent to fix the underlying issue.

Why it matters: Test failures that take hours to triage create pressure to disable tests rather than fix them. A platform that produces actionable failure output — which step failed, what was expected, what was found, screenshots, root cause hypothesis — dramatically reduces MTTR.

How to benchmark it:

For the last 20 test failures in your current system, measure: time from failure detected to failure resolved. Then run the same measurement against the agentic platform during the PoC. The reduction in MTTR is your productivity gain.

Platforms with AI-generated failure summaries typically outperform those with raw stack traces and screenshots alone. The goal is a failure report that gives the agent or developer enough context to begin fixing without re-running the test manually.

Running a Structured Agentic QA Benchmark PoC

A 30-day PoC structured around these five metrics gives you defensible data for vendor selection:

Week	Activity	Metrics Collected
1	Baseline measurement of current state	Maintenance hours, CI pass rate, coverage count
2	Onboard platform, migrate or generate initial tests	Setup friction, time-to-first-test
3	Run UI change battery (3 severity levels)	Heal rate by severity
4	Normal sprint with agent-generated PRs	CI pass rate, coverage velocity, MTTR

At the end of week 4, compare all five metrics against your baseline. If the platform does not show measurable improvement on at least three of the five metrics, it is not delivering on the agentic QA promise.

For enterprise-specific evaluation criteria — compliance, RBAC, audit logs, SLA — see the enterprise agentic QA checklist. For a comparison of the leading platforms on these dimensions, see best agentic QA tools in 2026.

Frequently Asked Questions

What is the most important benchmark metric for agentic QA?

Self-heal rate under real UI change is the most differentiating metric because it directly drives long-term maintenance cost. Tools with high heal rates sustain value over time; tools with low heal rates shift maintenance burden back to the team. Measure it on your actual application with real UI changes, not on vendor-provided demos.

How long should an agentic QA benchmark PoC run?

Four weeks minimum, 8 weeks ideally. The first two weeks are dominated by setup effects — onboarding friction, initial test generation, cache warming. Weeks 3–4 show steady-state performance. An 8-week PoC captures enough sprint cycles to measure CI pass rate stability meaningfully.

Can you benchmark agentic QA without running a full PoC?

Partially. You can assess heal rate by running a structured UI change battery in a short trial. You cannot reliably measure CI pass rate stability or maintenance burden without a longer trial on your actual application. Vendor-provided benchmarks and demo environments are not a substitute for measuring against your specific stack and UI.

What is a good self-heal rate for an agentic QA platform?

For minor UI changes (class renames, label changes): 90%+ is achievable. For moderate restructuring (component hierarchy changes): 70–85% with intent-based healing, 40–60% with rule-based fallback. For major redesigns (full section overhaul): 60%+ with intent-based systems is good. Below 40% on moderate restructuring means the maintenance burden will compound at scale.

How does agentic QA benchmark differently than traditional test automation?

Traditional test automation benchmarks focus on authoring speed, browser coverage, and integration compatibility — static properties measured at a point in time. Agentic QA benchmarks must measure dynamic properties: how the platform performs as the application evolves. Heal rate, CI stability over time, and coverage growth velocity are the metrics that matter, and they require time-boxed trials to measure accurately.

How to Detect Hidden Bugs in AI-Generated Code (2026)

Shiplight — Mon, 13 Apr 2026 02:11:51 +0000

AI coding agents ship code fast. That is the point. But speed without verification creates a specific failure mode: hidden bugs that pass linting, type checks, and even unit tests — but break under real user conditions. A checkout flow that works in dev fails in Safari. An auth edge case silently drops users. A refactored component breaks a flow three screens away.

Studies consistently show that AI-generated code has 1.7x more bugs than carefully reviewed human code. The issue is not that the models are incompetent — it is that the verification step has not kept pace with the generation step. AI generates code faster than any human can review it end-to-end, and most teams have not yet built the detection layer to close that gap.

This guide covers the specific techniques that catch hidden bugs in AI-generated code before users find them.

Why Hidden Bugs Are a Specific AI Code Problem

Traditional code review scales with the size of the diff. A developer writing 50 lines of code produces a 50-line PR that a reviewer can meaningfully evaluate. An AI coding agent implementing a feature across five files produces a 500-line diff in minutes — and the reviewer can approve it in seconds without actually verifying the behavior.

The bugs that survive this process are not syntax errors or obvious logic mistakes — those get caught by static analysis. The hidden bugs are:

Edge case failures: the agent implemented the happy path correctly but did not account for empty states, network failures, or invalid input
Cross-browser inconsistencies: CSS and JavaScript that behaves correctly in Chrome but fails in Firefox or Safari
Regression side effects: the agent changed a shared component and broke a flow it did not explicitly modify
Integration failures: a feature that works in isolation fails when combined with real authentication, session state, or live data
Silent failures: code that runs without errors but produces wrong outputs — the most dangerous category

These bugs have one thing in common: they require running the application in a real environment to detect. No static analysis tool catches a Safari layout regression. No unit test catches a state management bug that only appears after a user has navigated through three screens.

Detection Technique 1: Live Browser Verification on Every Agent Commit

The most direct way to detect hidden bugs in AI-generated code is to run the application in a real browser immediately after the agent commits. Not in CI — during development, before the code is even pushed.

Shiplight's browser MCP server enables this for any MCP-compatible agent (Claude Code, Cursor, Codex). After implementing a feature, the agent can:

Open the application in a real Playwright-powered browser
Navigate through the new feature end-to-end
Assert that expected elements are present and behave correctly
Capture screenshots as verification evidence
Flag any failures back to the developer before the PR is opened

This catches the largest category of hidden bugs — integration failures that are invisible in code review — at the point when they are cheapest to fix: before the diff leaves the developer's machine.

Detection Technique 2: Intent-Based E2E Regression Tests

One-time browser verification catches bugs at implementation time. Regression tests catch bugs that future agent commits introduce in code that was previously working.

The key design decision is how tests express what they are verifying. Tests written against specific DOM selectors (#checkout-btn, .form__total, data-testid="submit") break constantly as the agent refactors components. Tests written against user intent survive refactors because the intent does not change when the implementation does.

goal: Verify checkout flow completes for logged-in user
base_url: https://app.example.com
statements:
  - URL: /cart
  - intent: Click Proceed to Checkout
  - intent: Confirm shipping address is pre-filled
  - intent: Click Place Order
  - VERIFY: Order confirmation is displayed with order number

When the agent restructures the checkout component, this test does not need to be updated — the steps describe what the user does, not which CSS class the button currently has. The intent-cache-heal pattern resolves the correct element automatically when a cached locator becomes stale.

For teams using AI coding agents, this is the sustainable approach: tests that grow with the codebase without becoming a maintenance burden that requires its own engineering effort.

Detection Technique 3: Automated Regression Gates on Pull Requests

A test suite that runs manually is a test suite that gets skipped. The detection layer for AI-generated code needs to run automatically on every pull request, blocking merges when regressions are found.

The critical properties of an effective regression gate:

Runs on every PR, not on a schedule — regressions should be caught at the commit that introduces them, not discovered later
Blocks merge on failure — advisory-only results get ignored under shipping pressure
Provides actionable failure output — the agent needs to know which step failed, what was expected, and what was found, so it can diagnose and fix without human intervention

name: E2E Regression Gate
on:
  pull_request:
    branches: [main, staging]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run regression suite
        uses: shiplight-ai/github-action@v1
        with:
          api-token: ${{ secrets.SHIPLIGHT_TOKEN }}
          suite-id: ${{ vars.SUITE_ID }}
          fail-on-failure: true

When this gate is in place, AI coding agents receive structured failure output and can diagnose and fix regressions before the PR reaches human review. This creates the AI-native QA loop: the agent writes code, the gate catches regressions, the agent fixes them — without waiting for a human to click through the feature.

See E2E testing in GitHub Actions for a complete setup guide.

Detection Technique 4: Cross-Browser and Edge Case Coverage

AI coding agents are trained predominantly on code that targets the most common browser and environment configurations. Edge cases are underrepresented in the training data and underspecified in the prompts. This produces a predictable bug distribution: happy path in Chrome works, everything else is uncertain.

A detection strategy for AI-generated code should explicitly cover:

Cross-browser execution:

Run regression tests against Chromium, Firefox, and WebKit (Safari) automatically
Flag browser-specific failures separately so they can be triaged by affected audience
Pay particular attention to CSS layout, form behavior, and JavaScript API compatibility

Edge case scenarios:

Empty states: what happens when there is no data to display?
Error states: what happens when an API call fails?
Boundary conditions: maximum input lengths, minimum/maximum values, zero quantities
Concurrent actions: what happens if a user double-clicks a submit button?

User journey combinations:

Test flows that the agent did not explicitly implement — what happens to adjacent features?
Test with real session state (logged-in users, different role permissions, expired tokens)

These scenarios are underrepresented in agent-generated tests because the agent optimizes for the specified requirement. The detection layer needs to explicitly cover the space the agent did not think to test.

Detection Technique 5: AI-Powered Failure Analysis

Detecting that a bug exists is half the problem. The other half is diagnosing it fast enough that the fix happens in the same development session — not a week later when the context is cold.

Modern AI test platforms generate structured failure summaries that go beyond "step 3 failed." A useful failure summary includes:

Which step failed and why — not just the error message, but what was expected vs. what was found
Screenshot context — what the browser showed at the point of failure
Root cause hypothesis — is this a locator failure (UI changed) or a behavioral failure (application broke)?
Suggested fix direction — enough context for the agent to start diagnosing without re-running the test manually

Shiplight's AI Test Summary provides this output automatically on every test failure, reducing the time from "something failed" to "we know why and who fixes it" — which matters particularly when AI agents are processing multiple PRs simultaneously.

Building Your Detection Stack

The detection techniques above layer on each other. A practical implementation sequence:

Phase	Technique	Catch Rate
1	Live browser verification during development	Integration failures, layout bugs
2	Intent-based E2E regression suite	Behavioral regressions, edge cases
3	Automated PR gate	Regressions on every commit
4	Cross-browser coverage	Browser-specific bugs
5	AI failure analysis	Fast diagnosis and fix loop

Start with Phase 1 and 3 — browser verification during development and a blocking CI gate. These two steps catch the largest categories of hidden bugs with the least setup overhead. Add coverage depth as the agent generates more features.

Frequently Asked Questions

What types of bugs does AI-generated code most commonly hide?

The most common hidden bugs in AI-generated code are: edge case failures (empty states, error states, boundary conditions), cross-browser inconsistencies (CSS layout and JavaScript behavior), regression side effects (changes to shared components breaking adjacent flows), and silent failures (code that runs without errors but produces wrong outputs). These require runtime verification to detect — static analysis misses all of them.

Can unit tests catch hidden bugs in AI-generated code?

Unit tests catch logic errors in isolated functions but miss integration bugs, browser-specific behavior, and regression side effects. A function that correctly processes a payment object in isolation may still fail in the context of a real checkout flow with authentication, session state, and API calls. End-to-end browser tests are required to catch the hidden bug categories that AI-generated code is most prone to.

How do you test AI-generated code without slowing down the development loop?

The key is running verification at two points: immediately after implementation (browser verification during development via MCP), and automatically on every PR (CI gate). The first catches bugs before they are pushed. The second catches regressions before they merge. Both are automated — the developer does not manually run tests on every change.

What is the best way to write tests for code that changes frequently?

Write tests against user intent rather than DOM selectors. An intent-based test ("click the submit button", "verify the confirmation message") remains valid when the agent renames classes, restructures components, or refactors the implementation. Selector-based tests break on every refactor. See what is self-healing test automation for a full explanation of how intent-based healing works.

How does browser verification differ from unit testing for AI code?

Browser verification runs the actual application in a real browser and simulates real user interactions — clicking buttons, filling forms, navigating between pages. It catches bugs that unit tests cannot: layout regressions, cross-browser inconsistencies, integration failures between components, and behavioral bugs that only appear in the context of a full user journey.

Test Harness Engineering for AI Test Automation (2026 Guide)

Shiplight — Mon, 13 Apr 2026 02:11:15 +0000

A test harness is the infrastructure layer that surrounds your tests: the fixtures, configuration, environment management, data setup, and execution scaffolding that make individual tests runnable, repeatable, and meaningful. In traditional testing, building a good harness is an engineering discipline in its own right. In AI test automation, it is the critical differentiator between a fragile prototype and a production-grade quality system.

As AI coding agents accelerate feature delivery, the harness needs to keep pace. This guide covers the core techniques for test harness engineering that work with AI test automation — not against it.

What Is a Test Harness?

A test harness is everything that is not the test itself. It includes:

Fixtures: reusable setup and teardown routines (authenticated sessions, seed data, environment state)
Configuration layer: environment URLs, credentials, feature flags, and runtime parameters
Execution driver: the runtime that interprets and runs test definitions (Playwright, pytest, a custom runner)
Reporting pipeline: how results flow to CI, dashboards, and alerting systems
Self-healing layer: how the harness handles locator failures without requiring manual intervention

In manual testing, the harness is implicit — testers carry this context in their heads. In automated testing, the harness is explicit and must be maintained as carefully as the tests themselves. In AI test automation, where tests are generated at machine speed and the application changes frequently, the harness design determines whether your test suite grows sustainably or collapses under its own weight.

Why Traditional Harnesses Break with AI-Generated Code

Traditional test harnesses are built around a stable, human-paced development cycle. The harness assumes:

Selectors are stable enough to hard-code or record
Component structure changes infrequently enough to update manually
Test data setup scripts can be maintained by whoever wrote them
One person understands the full harness context

AI coding agents break all four assumptions. An agent refactors a component in minutes, renames classes across files, and restructures DOM hierarchies as a side effect of implementing an unrelated feature. Tests that depend on #submit-btn or .checkout-form__total fail constantly — not because the application broke, but because the locator cache is stale.

The result: teams either cap their test suites at a size they can manually maintain, or they accept a permanent background noise of broken tests that get disabled rather than fixed. Neither outcome is acceptable for teams shipping at AI speed.

Harness Engineering Technique 1: Intent-Based Test Definitions

The most important structural decision in a modern test harness is how tests express what they are testing. Traditional harnesses store locators as the source of truth. Intent-based harnesses store the user goal as the source of truth and treat locators as a derived, cached artifact.

In practice, this means each test step describes what a user is doing — not how the DOM is currently structured:

goal: Verify checkout flow completes successfully
base_url: https://app.example.com
statements:
  - URL: /cart
  - intent: Click the Proceed to Checkout button
  - intent: Fill in shipping address with test data
  - intent: Select standard shipping
  - intent: Click Place Order
  - VERIFY: Order confirmation number is visible

When the UI changes — a button moves, a class renames, a container restructures — the intent remains valid. The harness resolves the correct element against the current page state rather than failing on a stale selector. This is the foundation of the intent-cache-heal pattern: intent as the authoritative definition, cached locators for execution speed, AI resolution when the cache misses.

Harness Engineering Technique 2: Declarative Configuration in Version Control

A test harness that lives outside version control is a harness you cannot trust, audit, or reproduce. The configuration layer — environment URLs, test suites, execution parameters — should live in your repository alongside application code.

YAML-based test configuration makes this natural. Each test file is a human-readable YAML document that specifies the goal, the base URL, and the sequence of user actions. The harness configuration is a separate YAML file that references these test files and defines execution parameters:

suite: checkout-regression
environment: staging
base_url: https://staging.example.com
tests:
  - tests/checkout/full-flow.yaml
  - tests/checkout/guest-checkout.yaml
  - tests/checkout/promo-code.yaml
parallelism: 4
fail_fast: false

This approach gives you several properties that matter at scale:

Auditability: every change to test definitions and configuration is visible in git history
Portability: no vendor lock-in — the test definitions are readable without the platform
Ownership: whoever owns the feature owns the tests — the YAML lives next to the application code
Reproducibility: any CI environment can run the same configuration deterministically

Harness Engineering Technique 3: Self-Healing Locator Cache

Speed and resilience are usually in tension in test harnesses. Fast tests use cached locators. Resilient tests use AI resolution. A well-designed harness does not choose — it uses both, with a fallback strategy.

The pattern:

First run: AI resolves the element from the intent description and caches the locator
Subsequent runs: the cached locator is used directly — execution is as fast as any Playwright test
Cache miss: the locator fails because the UI changed. The harness falls back to AI resolution using the original intent, finds the new element, and updates the cache
Cache update: on the next run, the resolved locator is used again

This architecture means the harness is deterministic and fast in the common case (the UI has not changed) and resilient in the edge case (the UI has changed). The self-healing layer is invoked rarely, keeping execution speed predictable.

For AI-driven development workflows, where the application changes on every agent commit, this is the only sustainable approach. See self-healing vs. manual maintenance for a detailed comparison of the maintenance burden across approaches.

Harness Engineering Technique 4: Fixture Isolation for AI-Generated Tests

AI coding agents generate tests rapidly, but they do not have visibility into shared fixture state. A naive harness lets tests share mutable state: one test logs in, creates a record, and leaves it for the next test. This works until two tests run in parallel and corrupt each other's state.

Robust harness engineering for AI test automation requires fixture isolation:

Session isolation: each test run gets a fresh authenticated session, not a shared one
Data isolation: test data is created per-test and cleaned up after — or tests use stable seed data that is never mutated
Environment isolation: parallel test runs target separate environment instances or use per-test namespacing to avoid collisions

For authentication specifically, the most reliable pattern is to log in once per test run, save the session state, and reuse it across tests in that run — without re-authenticating on every step. Shiplight's harness supports session state persistence out of the box, which is particularly important for testing SSO, 2FA, and magic link flows.

Harness Engineering Technique 5: CI Gate Integration as a Harness Contract

A test harness is only valuable if its results are actionable. The final layer of harness engineering is integrating execution results into your CI pipeline as a blocking gate — not an advisory report.

The harness should:

Run on every pull request, including those generated by AI coding agents like Codex or Claude Code
Report pass/fail as a required status check that blocks merge on failure
Surface failure context — which step failed, what was expected, what was found, with screenshots — so the agent or developer can act immediately without context switching

GitHub Actions integration for a YAML-based harness looks like this:

name: E2E Regression Suite
on:
  pull_request:
    branches: [main, staging]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run E2E harness
        uses: shiplight-ai/github-action@v1
        with:
          api-token: ${{ secrets.SHIPLIGHT_TOKEN }}
          suite-id: ${{ vars.SUITE_ID }}
          fail-on-failure: true

When an AI coding agent opens a PR that breaks a test, the CI gate catches it. The agent receives the structured failure output and can diagnose and fix the issue before the PR reaches human review. This closes the AI-native QA loop: write, verify, gate, fix — without waiting for a human to click through the feature.

Building the Harness Incrementally

A complete test harness does not need to be built all at once. The practical sequence:

Start with one critical flow in an intent-based YAML file — signup, checkout, or core authentication
Add it to CI as a required check on the branch that touches that flow
Expand coverage as the agent generates new features — add tests alongside the code
Introduce fixture isolation when parallel execution becomes necessary
Add scheduling for continuous execution against production

Each step adds value independently. A single self-healing test wired into CI is more valuable than a comprehensive suite that runs manually on a schedule.

Frequently Asked Questions

What is the difference between a test harness and a test framework?

A test framework provides the primitives for writing and running tests (assertions, test runners, reporters). A test harness is the application-specific layer built on top: the fixtures, configuration, authentication helpers, and execution infrastructure specific to your application. Playwright is a framework. The YAML configuration, session fixtures, and CI integration that surround your Playwright tests are the harness.

How does intent-based testing improve harness maintainability?

Intent-based tests define what the user is doing rather than which DOM element to interact with. When the UI changes — a class renames, a component restructures, a button moves — the intent remains valid and the harness resolves the correct element automatically. This eliminates the most common source of harness maintenance: updating stale selectors after UI changes.

How should a test harness handle AI-generated code that changes frequently?

Two techniques: self-healing locators that resolve from intent when the cached locator fails, and intent-based test definitions that remain valid through UI restructuring. Together, these mean the harness does not need to be updated every time the agent refactors a component. The intent-cache-heal pattern is the practical implementation of both.

Can the same harness work for both human-written and AI-generated tests?

Yes. Intent-based YAML test files can be authored by humans, generated by AI agents, or produced by a combination. The harness executes them identically. This is important for teams that use AI agents to generate initial test coverage and then refine tests manually.

What CI/CD pipelines does a YAML test harness support?

A well-designed harness should support GitHub Actions, GitLab CI, Azure DevOps, and CircleCI through standard API-based triggers. Shiplight's harness integration works with all four through either a native GitHub Action or API-based triggers for other pipelines.

How to QA Code Written by Claude Code

Shiplight — Sat, 11 Apr 2026 05:46:05 +0000

Claude Code is fast. Give it a well-formed prompt, and it will write a working implementation, refactor your components, fix a failing test, and open a pull request — all without leaving your terminal. For teams that have adopted it, the productivity gain is measurable within a week.

The gap is verification. Claude Code is optimized for writing code, not for confirming that the code works end-to-end in a real browser across the full feature surface. That step still defaults to a human clicking through the UI manually, or to a test suite that may not exist yet.

This guide covers how to close that gap: giving Claude Code the tools to verify its own work, capture those verifications as regression tests, and ship with confidence.

Why Claude Code Needs a QA Layer

Claude Code operates within your terminal and editor. It reads files, writes files, runs commands, and navigates your codebase. What it cannot do by default is open a browser, interact with your live application, and observe whether the UI behaves correctly.

This matters more than it might seem. A significant portion of frontend bugs are not logic errors — they are integration failures: a component that renders correctly in isolation but breaks when combined with real data, a form that passes validation in unit tests but submits incorrectly in the browser, an animation that works in Chrome but fails in Safari.

Claude Code will not catch these without a browser. And if you are relying on your own manual verification to catch them, you are creating a quality bottleneck that scales inversely with how fast your agent ships.

The solution is to extend Claude Code's toolchain with browser access — so the agent can verify its own work before it asks you to review a pull request.

Setting Up the Shiplight MCP Server with Claude Code

Shiplight's browser MCP server gives Claude Code a real browser it can control during development. Once configured, Claude Code can open your application, navigate through features it just built, and confirm they work — autonomously.

Installation

Add the Shiplight MCP server to your Claude Code configuration:

{
  "mcpServers": {
    "shiplight": {
      "command": "npx",
      "args": ["-y", "@shiplight/mcp"]
    }
  }
}

No account is required to get started. The MCP server connects Claude Code to a local browser instance that it can automate using Shiplight's browser tools.

What Claude Code Can Do with the Browser

Once the MCP server is active, you can instruct Claude Code to:

Open your application in a real browser and navigate to a specific feature
Interact with the UI — fill forms, click buttons, trigger flows
Verify assertions — confirm that text appears, elements are present, redirects work
Capture screenshots as evidence of successful verification
Save verifications as YAML tests that run automatically in CI

A typical instruction looks like: "Implement the new onboarding flow, then verify it end-to-end in the browser and save the verification as a test."

Claude Code handles the implementation and the verification. You review the evidence — screenshots, test file, and CI results — rather than clicking through the feature yourself.

Generating Self-Healing Tests from Claude Code Verifications

Manual browser verification is valuable, but ephemeral. The real leverage is when those verifications become permanent regression tests.

Shiplight uses a YAML test format where each step is expressed as an intent rather than a DOM selector:

goal: Verify onboarding flow completes successfully
base_url: https://app.example.com
statements:
  - URL: /signup
  - intent: Enter a valid email address in the signup form
  - intent: Click the "Get Started" button
  - VERIFY: Welcome screen is visible with the user's name

Claude Code can generate these files directly after verifying a feature. Instruct it to: "After verifying the onboarding flow, save the browser steps as a Shiplight YAML test in the tests/ directory."

The tests are written against intent, not implementation details. When Claude Code refactors a component, the tests adapt rather than break — because the intent (what the user is doing) has not changed, only the DOM structure.

This is the key insight behind the intent-cache-heal pattern: tests that survive the pace of AI-driven development.

Running Tests in CI on Every Claude Code Pull Request

Once Claude Code is generating YAML tests, the next step is running them automatically on every pull request.

Shiplight integrates with GitHub Actions so your test suite runs as a CI check on every PR. If Claude Code's changes break an existing flow, the PR is flagged before merge.

A minimal GitHub Actions configuration:

name: E2E Tests
on: [pull_request]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run Shiplight tests
        uses: shiplight-ai/github-action@v1
        with:
          api-token: ${{ secrets.SHIPLIGHT_TOKEN }}
          suite-id: ${{ vars.SUITE_ID }}

With this in place, Claude Code's workflow completes a full loop: implement → verify in browser → generate test → CI gates the merge. You get the speed of an AI coding agent with the quality guarantees of a test suite.

Best Practices for Claude Code QA

Be explicit about verification in your prompts

Claude Code will verify its work if you ask it to. Include verification as part of your task descriptions:

✅ "Implement the billing settings page. After implementing, verify it works in the browser and generate a test."
❌ "Implement the billing settings page."

Verification does not happen automatically unless the MCP server is active and the prompt includes it.

Scope tests to user journeys, not implementation details

Ask Claude Code to test what the user does, not what the code does. Tests tied to user actions survive future refactors; tests tied to specific component names or class names do not.

Review the test file, not just the feature

When Claude Code generates a YAML test, read it. The test is documentation of what was verified and how. If the test only covers the happy path, prompt Claude Code to add edge cases: "Add test cases for validation errors and network failure states."

Use the Shiplight VS Code extension for debugging

If a test fails, the Shiplight VS Code extension lets Claude Code step through the test interactively — seeing exactly what the browser shows at each step. Claude Code can diagnose and fix failures without you needing to reproduce them manually.

What Gets Verified vs. What Still Needs Human Review

A QA-enabled Claude Code workflow handles the bulk of verification automatically, but some things still benefit from human judgment:

Automated by Shiplight	Human review still valuable
Feature works end-to-end	Visual design and UX quality
Existing flows not regressed	Business logic edge cases you haven't specified
Cross-browser behavior	Accessibility beyond automated checks
CI gate on PRs	Security-sensitive flows

The goal is not to eliminate human review — it is to ensure that by the time something reaches human review, the mechanical correctness is already confirmed.

Frequently Asked Questions

Does Shiplight replace Claude Code's built-in browser tools?

Shiplight extends Claude Code's capabilities rather than replacing them. The MCP server adds browser automation, test generation, and CI integration on top of what Claude Code already does. It is an additional tool in the agent's toolchain.

Can Claude Code write tests without a browser MCP server?

Claude Code can write unit tests and integration tests without a browser. For E2E tests that verify real user journeys in a live application, a browser MCP server is required.

How does Shiplight handle authentication in tests?

Shiplight supports persistent browser profiles and authentication flows, including email-based login and OAuth. Tests can be set up to authenticate before running scenarios. See the authentication testing guide for details.

Are the YAML test files compatible with existing Playwright setups?

Yes. Shiplight runs on top of Playwright and its YAML tests coexist with standard Playwright test files. You can adopt YAML tests incrementally without migrating your existing test suite.

What if Claude Code's test does not cover an edge case I care about?

After Claude Code generates a test, you can edit the YAML file to add additional steps, or prompt Claude Code: "Add a test case for [specific scenario]." The YAML format is designed to be readable and editable by both humans and AI.

References: Claude Code documentation, Playwright Documentation, Shiplight MCP Documentation

OpenAI Codex Testing: How to QA AI-Written Code

Shiplight — Sat, 11 Apr 2026 05:45:06 +0000

OpenAI Codex is an autonomous coding agent that can take a task, implement it across your codebase, and produce a pull request — without a developer writing a line of code. For engineering teams, that is a significant acceleration. For QA teams, it raises an immediate question: who verifies what Codex wrote?

The honest answer for most teams: nobody, systematically. Codex generates code faster than any human can review it end-to-end. Manual verification does not scale. And most teams have not yet built the automated QA layer that would catch what Codex misses.

This article covers how to build that layer — a testing workflow that keeps pace with Codex's output, catches regressions before they reach production, and does not create a new maintenance burden every time Codex refactors something.

The Quality Challenge with AI-Generated Code

AI coding agents like Codex are optimized for producing syntactically correct, functionally reasonable code based on the task specification. They are not optimized for:

Edge cases not mentioned in the prompt — Codex implements what you asked for, not everything that could go wrong
Cross-browser compatibility — generated CSS and JavaScript may behave differently across browser engines
Interaction with existing code — Codex changes may introduce unexpected behavior in adjacent features it did not directly modify
Real-world user flows — a feature that works in isolation may fail when combined with authentication, real data, or specific browser states

Research consistently shows that AI-generated code introduces bugs at higher rates when the verification loop is truncated. The issue is not that Codex writes bad code — it is that the review step cannot keep pace with the generation step without tooling support.

What a Codex QA Workflow Needs

An effective QA workflow for Codex-generated code has three components:

Live browser verification — test the actual running application, not just the code in isolation
Regression coverage — ensure Codex's changes did not break existing functionality
Automatic test generation — capture verifications as persistent tests without manual test authoring

Each component addresses a specific failure mode. Browser verification catches integration bugs that unit tests miss. Regression coverage catches unintended side effects. Automatic test generation ensures the coverage grows with the codebase without creating a maintenance backlog.

Browser Verification for Codex Output

The most direct way to verify Codex output is to run the application and interact with the new feature the way a user would.

Shiplight's browser MCP server enables this for any MCP-compatible agent. After Codex implements a feature, an AI agent with MCP access can:

Open the application in a real browser
Navigate to the new feature
Execute the user journey end-to-end
Assert that the expected outcomes are present
Capture screenshots as verification evidence

This happens within the same development loop — no context switch to a separate testing environment. The verification step becomes part of how the feature gets built, not a separate phase after it.

For teams using Codex alongside other agents (Claude Code, Cursor, or custom orchestration), the Shiplight MCP server integrates with any tool that supports the Model Context Protocol.

Generating Self-Healing Tests from Codex Verifications

One-time browser verification catches bugs at the point of implementation. Persistent regression tests catch bugs that future changes introduce.

Shiplight converts browser verifications into YAML test files that live in your repository and run automatically in CI. Each test step is expressed as a user intent rather than a DOM locator:

goal: Verify task creation flow works end-to-end
base_url: https://app.example.com
statements:
  - URL: /dashboard
  - intent: Click "New Task" to open the task creation dialog
  - intent: Enter a task title and assign it to a team member
  - intent: Click "Create Task"
  - VERIFY: New task appears in the dashboard task list

This format is critical for Codex workflows specifically. Codex frequently refactors component structure, renames classes, and reorganizes DOM hierarchies as part of implementation. Tests written against specific CSS selectors break constantly. Tests written against user intent — what the user is doing, not how the DOM is currently structured — survive refactors because the intent does not change when the implementation does.

This is the intent-cache-heal pattern: intent as the source of truth, cached locators for speed, AI resolution when the cache is stale. It is the only testing approach that keeps pace with agents that change your UI frequently.

Setting Up CI Gates for Codex Pull Requests

The final step is making the test suite a blocking check on every Codex pull request. Without a CI gate, tests are advisory. With one, Codex cannot merge code that breaks an existing user flow.

Shiplight integrates with GitHub Actions for automatic test execution on pull requests:

name: E2E Regression Tests
on:
  pull_request:
    branches: [main, staging]

jobs:
  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Run E2E suite
        uses: shiplight-ai/github-action@v1
        with:
          api-token: ${{ secrets.SHIPLIGHT_TOKEN }}
          suite-id: ${{ vars.SUITE_ID }}
          fail-on-failure: true

When a Codex PR breaks a test, GitHub flags the PR as failed. The agent receives the failure output and can diagnose and fix the issue before the PR reaches human review.

This closes the Codex quality loop: the agent implements, verifies, generates tests, and responds to CI failures — all without waiting for a human to click through the feature manually.

Handling High-Velocity Codex Output

Teams using Codex for autonomous development often have multiple PRs open simultaneously. A QA workflow for this environment needs to handle:

Parallel test runs — multiple PRs running tests concurrently without blocking each other. Shiplight Cloud handles parallel execution without additional configuration.

Test suite growth — as Codex adds features, the test suite grows. YAML templates allow common sequences (login, navigation, data setup) to be defined once and reused across tests, preventing the suite from becoming thousands of one-off scripts.

Failure triage — when multiple PRs fail tests, engineering teams need to understand which failures are real regressions vs. expected changes. Shiplight's AI Test Summary analyzes failure output and provides root-cause context, reducing the time from "something failed" to "we know why and who owns it."

Codex Testing: What to Automate vs. What to Review Manually

Automate with Shiplight	Review manually
Critical user journeys (signup, login, checkout, key settings)	Visual design quality
Regression across existing features	Business logic correctness for new requirements
Cross-browser behavior	Security-sensitive flows
CI gate on Codex PRs	Accessibility audits
Evidence capture (screenshots, step logs)	Final production approval

The goal is not to eliminate human judgment — it is to ensure that by the time a Codex PR reaches human review, you know it does not break anything that was already working. That frees reviewers to focus on whether the implementation is correct for the requirement, not on whether it accidentally broke the login flow.

Frequently Asked Questions

What is OpenAI Codex and how does it differ from ChatGPT?

OpenAI Codex is an autonomous coding agent designed to implement software tasks end-to-end — reading your codebase, writing code, running tests, and opening pull requests. ChatGPT generates conversational responses. Codex is optimized for code generation and repository-level task execution.

Can Codex write its own tests?

Codex can write unit tests and sometimes integration tests as part of its implementation. For end-to-end browser tests that verify real user journeys, Codex needs browser access via an MCP server and a test format that survives frequent UI changes. Shiplight provides both.

How do self-healing tests work with Codex's frequent refactors?

Self-healing tests use AI to resolve user intent against the current page state when a cached locator fails. If Codex restructures a component, the test finds the correct element by matching its semantic description rather than a specific CSS selector. See What Is Self-Healing Test Automation for the full explanation.

Does this work with Codex's GitHub integration?

Yes. Codex submits pull requests to GitHub. Shiplight's GitHub Actions integration runs tests automatically on those pull requests and reports pass/fail status as a PR check — the same as any other CI workflow.

How do I handle tests for features that change frequently during Codex development?

Write tests at the user journey level, not the implementation level. If a test describes "user can create a project and invite a collaborator," it will stay valid through UI changes. If it describes "click the element with id='project-create-btn'", it will break every time Codex refactors the component.

References: OpenAI Codex documentation, Playwright Documentation, GitHub Actions documentation

Vibe Coding Is Fun Until Production Breaks

Shiplight — Sat, 11 Apr 2026 05:45:05 +0000

Vibe coding is exactly what it sounds like: you describe what you want, your AI coding agent writes the implementation, and you ship it. No wrestling with boilerplate, no context-switching into unfamiliar APIs, no debugging stack traces line by line. Just intent → code → deploy.

It is genuinely fast. Teams that have adopted AI-first development workflows report shipping features in hours that previously took days. The experience is intoxicating.

The problem shows up in production. Not always immediately, not always dramatically — but consistently. A checkout flow that worked in the demo breaks for users in a specific browser. An edge case in the new auth logic causes silent failures. A UI component that the agent refactored now behaves differently when the viewport changes. The AI wrote correct code for the happy path, but nobody verified the full surface area.

This is the vibe coding quality gap: the speed gain is real, but the verification step got left out.

What Vibe Coding Actually Skips

Traditional software development has a built-in quality loop. Developers write code, run tests, review diffs, and iterate before shipping. Each step adds friction — but that friction catches bugs.

Vibe coding compresses this loop dramatically. The agent writes the code, you review a high-level summary, and the diff goes out. The problem is that the review step scales poorly with the agent's output. A human can meaningfully review 50 lines of code. Reviewing 500 lines of agent-generated implementation across five files is a different task entirely.

What actually gets skipped in most vibe coding workflows:

End-to-end verification — does the feature actually work from a user's perspective?
Regression coverage — did the agent's changes break something it wasn't supposed to touch?
Edge case validation — what happens with empty states, network failures, or unexpected inputs?
Cross-browser consistency — did the agent's CSS choices work everywhere?

These are not hypothetical concerns. Research on AI-generated code quality consistently shows that AI-written code introduces bugs at higher rates than carefully reviewed human code — not because the models are bad, but because the verification loop is truncated.

The Speed Trap

Here is the dynamic that makes vibe coding quality gaps compound over time.

When you ship fast and something breaks, the natural response is to have the agent fix it. The agent patches the bug, you ship the patch, and you move on. This works fine for isolated issues. But over weeks and months, an unverified codebase accumulates a debt of untested edge cases. Each fix potentially introduces new issues. The agent has no memory of what it previously changed or why.

Without a persistent test suite, you have no ground truth. You cannot tell whether the latest agent commit made things better or worse in aggregate. You only find out when a user reports something.

This is not a problem with the AI coding agents themselves — they are doing exactly what they were designed to do. It is a workflow design problem. The quality layer was never added.

Adding QA to Your Vibe Coding Workflow

The good news is that vibe coding and comprehensive testing are not in conflict. The same agents that write your application code can be directed to write tests, run verifications, and maintain a quality gate — if you give them the right tools.

Step 1: Give your agent a browser

The most immediate gap in vibe coding workflows is live browser verification. Your agent can write a component, but it cannot see what that component looks like or how it behaves without a browser.

Shiplight's browser MCP server gives your AI coding agent eyes and hands in a real browser. During development, the agent can open your application, navigate through the new feature, and verify that what it built actually works — before the code leaves your machine.

This closes the most common vibe coding failure mode: code that passes linting and type checks but fails in practice.

Step 2: Capture verifications as regression tests

Every time your agent verifies a feature in the browser, that verification can become a permanent test. Shiplight converts browser interactions into YAML test files that live in your repo and run automatically in CI.

These are not brittle tests that break every time your UI changes. The tests are written against the intent of each step ("Click the submit button", "Verify the confirmation message appears"), not against specific DOM selectors. When your agent makes future changes, the tests adapt rather than fail on superficial differences.

Step 3: Run tests on every agent commit

Once you have a test suite, wire it into your CI pipeline so every agent-generated commit gets verified before merge. Shiplight's GitHub Actions integration makes this a one-time setup.

The result: your agent can ship code at full vibe coding speed, and you get a regression gate that catches problems before they reach production.

The Intent-Cache-Heal Pattern for Vibe Coders

Traditional test automation breaks constantly because tests are tied to implementation details — specific CSS selectors, DOM structure, element IDs — that agents change freely. This is why most vibe coding teams do not bother with E2E tests: the maintenance burden exceeds the value.

The intent-cache-heal pattern solves this. Tests describe what the user is trying to accomplish, not how the UI is currently built. When your agent restructures a component, the test heals automatically because the intent has not changed — only the implementation.

This is the missing piece that makes comprehensive testing compatible with vibe coding's pace. You are not maintaining tests after every agent commit. The tests maintain themselves.

What a Vibe Coding + QA Workflow Looks Like

A practical workflow looks like this:

Describe the feature to your agent (Claude Code, Cursor, Codex, or any MCP-compatible agent)
Agent implements the feature and opens it in a real browser via the Shiplight MCP server
Agent verifies the feature works end-to-end and documents the verification as a YAML test
CI runs the test suite on the pull request — any regressions block the merge
Agent fixes flagged issues with the context from the test failure output
Merge with confidence — the full feature surface is verified

The agent handles steps 2 through 5. Your job is to define the intent and review the evidence. That is what vibe coding should feel like.

Frequently Asked Questions

What is vibe coding?

Vibe coding is a development style where developers use AI coding agents to write code by describing intent in natural language. The AI agent handles implementation while the developer focuses on what the product should do rather than how to build it.

Why does vibe coding produce bugs?

Vibe coding itself does not produce more bugs than traditional development — but the truncated review cycle means bugs are caught later. AI coding agents write for the specified requirements and may miss edge cases, cross-browser differences, or regressions in code they did not explicitly touch.

Can AI agents write their own tests?

Yes. With the right tooling, AI coding agents can generate tests automatically from their own verifications. Shiplight's MCP server lets agents verify features in a real browser and capture those verifications as self-healing YAML test files that live in your repo.

Does adding tests slow down vibe coding?

Not significantly, when tests are generated automatically by the agent rather than written by hand. The overhead is a one-time CI setup. After that, tests run in the background and only interrupt the workflow when a real regression is found.

How do self-healing tests work with frequently changing UIs?

Self-healing tests are written against the intent of each user action, not specific DOM selectors. When the UI changes, the test framework resolves the correct element by matching the described intent to the current page state. See What Is Self-Healing Test Automation for a full explanation.

References: Playwright Documentation, GitHub Actions documentation