YAML-Based Testing: A New Approach to E2E

End-to-end testing has a maintenance problem.

Traditional test scripts are brittle, verbose, and tightly coupled to the DOM. A single UI refactor can break dozens of tests that were working perfectly the day before. Teams end up spending more time fixing tests than writing new ones.

YAML-based testing takes a different approach. Instead of writing procedural scripts that describe how to interact with elements, you write declarative files that describe what you want to test. The execution engine handles the how.

This is not a theoretical concept — Shiplight uses YAML as its native test format, and the approach fundamentally changes how teams think about E2E test maintenance.

Why YAML for Testing

The choice of YAML is deliberate. It solves three problems that plague traditional E2E test scripts.

Readability. A YAML test file reads like a checklist of user actions. Anyone on the team — developers, QA engineers, product managers — can read a YAML test and understand what it covers. Playwright scripts require JavaScript knowledge and familiarity with the Playwright API. YAML requires knowing what your application should do.

Separation of intent from implementation. Traditional scripts mix test logic with DOM interaction code. When a button's selector changes, the test breaks even though the user intent has not changed. YAML-based tests separate what you want to test (the intent) from how the tool finds and interacts with elements (the cached locators).

Version control friendliness. YAML diffs are clean and meaningful. When a test changes, the diff shows exactly what behavior changed. JavaScript test diffs often include noise from selector updates, async handling changes, and framework boilerplate.

How YAML Tests Differ from Playwright Scripts

To understand the difference, compare the same test written both ways.

A Playwright script for testing a login flow:

const { test, expect } = require('@playwright/test');

test('user can log in and see dashboard', async ({ page }) => {
  await page.goto('https://app.example.com/login');
  await page.fill('[data-testid="email-input"]', '[e-mail]');
  await page.fill('[data-testid="password-input"]', 'securepass123');
  await page.click('[data-testid="login-button"]');
  await page.waitForURL('**/dashboard');
  await expect(page.locator('[data-testid="welcome-message"]'))
    .toContainText('Welcome back');
  await expect(page.locator('[data-testid="project-list"]'))
    .toBeVisible();
});

The same test as a YAML file:

name: User login and dashboard
url: https://app.example.com/login
statements:
  - action: FILL
    target: email input
    value: [e-mail]
  - action: FILL
    target: password input
    value: securepass123
  - action: CLICK
    target: login button
  - action: VERIFY
    assertion: page contains "Welcome back"
  - action: VERIFY
    assertion: project list is visible

The YAML version is shorter, but length is not the point. The structural differences matter more.

The Playwright script contains seven selectors ([data-testid="email-input"], etc.) that will break if the frontend team renames those test IDs. The YAML version uses intent targets like email input and login button — descriptions of what the element is, not how to find it.

The Playwright script requires knowledge of async/await, the Playwright API, and JavaScript destructuring. The YAML version requires knowing what your application does.

Intent Statements

The core concept in YAML-based testing is the intent statement. An intent statement describes what you want to happen without prescribing how the tool should accomplish it.

When you write target: login button, you are expressing intent: "I want to interact with the thing the user would identify as the login button." The testing engine resolves this to an actual DOM element using AI-powered element matching.

This is fundamentally different from a selector like button.btn-primary.auth-submit or even [data-testid="login-btn"]. Selectors are implementation details. Intents are user-facing descriptions.

The intent-cache-heal pattern makes this practical at scale:

1. The first time a test runs, the engine resolves each intent to a specific locator and caches it.
2. On subsequent runs, the cached locator is used directly for speed.
3. If the cached locator fails (because the UI changed), the engine re-resolves the intent using AI.

You get the speed of cached selectors with the resilience of intent-based matching.

Cached Locators

Under the hood, every intent target is backed by a cached locator. When Shiplight first resolves login button to button[type="submit"], it stores that mapping in a locator cache file alongside your test.

# .shiplight/cache/login-test.locators.yml
- intent: email input
  locator: 'input[name="email"]'
  resolved_at: 2026-03-28T14:22:00Z
- intent: password input
  locator: 'input[name="password"]'
  resolved_at: 2026-03-28T14:22:01Z
- intent: login button
  locator: 'button[type="submit"]'
  resolved_at: 2026-03-28T14:22:01Z

These cache files live in your repo — version-controlled, reviewable artifacts. When a locator heals (re-resolves after a UI change), the cache file updates, and the diff shows exactly what changed. You can see every locator, when it was last resolved, and how it has evolved.

This transparency matters for teams that need to audit their test infrastructure. Nothing is hidden.

VERIFY Assertions

YAML-based tests use VERIFY steps for assertions. Unlike traditional assertions that check specific DOM properties, VERIFY steps express what should be true about the page in plain language.

- action: VERIFY
  assertion: page contains "Welcome back"
- action: VERIFY
  assertion: project list shows at least 3 items
- action: VERIFY
  assertion: navigation menu is visible
- action: VERIFY
  assertion: error message is not displayed

The testing engine determines the appropriate DOM checks to perform. The assertion works regardless of how the UI framework renders the content — whether it's a <span>, a <p>, or a <div>. You describe the outcome, not the implementation.

A Complete YAML Test Example

Here is a full YAML test for an e-commerce checkout flow, showing the full range of available actions and assertions:

name: Complete checkout flow
url: https://store.example.com
tags:
  - checkout
  - critical-path
statements:
  - action: CLICK
    target: first product card
  - action: VERIFY
    assertion: product detail page is displayed
  - action: CLICK
    target: add to cart button
  - action: VERIFY
    assertion: cart badge shows "1"
  - action: CLICK
    target: cart icon
  - action: VERIFY
    assertion: cart contains 1 item
  - action: CLICK
    target: proceed to checkout
  - action: FILL
    target: shipping address
    value: 123 Test Street, San Francisco, CA 94102
  - action: FILL
    target: card number
    value: "4242424242424242"
  - action: FILL
    target: expiration date
    value: "12/28"
  - action: FILL
    target: CVV
    value: "123"
  - action: CLICK
    target: place order button
  - action: VERIFY
    assertion: order confirmation page is displayed
  - action: VERIFY
    assertion: page contains "Thank you for your order"
  - action: VERIFY
    assertion: order number is displayed

This test will likely survive a complete frontend redesign as long as the checkout flow itself does not change. The equivalent Playwright script would be 60–80 lines of JavaScript with selectors, waits, and assertions tightly coupled to the current DOM.

Getting Started

If you are currently writing Playwright scripts, you do not need to rewrite everything at once. Shiplight runs alongside your existing test suite through its plugin system.

A practical starting point: identify your most-maintained tests — the ones that break frequently due to UI changes. Convert those to YAML format first and run them in parallel with your existing scripts. You will quickly see whether the maintenance burden drops.

For teams generating tests with AI coding agents, YAML is the natural output format. AI-generated JavaScript test scripts are hard to review and expensive to maintain. AI-generated YAML test files read like specs and stay readable over time.

---

References: Playwright Documentation · YAML Specification · Shiplight Plugins