DEV Community: Denis Skvortsov

17 PRs a Day, One QA: How We Automated E2E Failure Triage

Denis Skvortsov — Wed, 15 Jul 2026 13:30:13 +0000

Picture this: 18 AI-augmented developers merging an average of 17 pull requests every working day — 33 on the peak day. And at the time, exactly one QA engineer watching over all of it. The fact that things didn't fall apart daily looks like a miracle. Spoiler: it's not a miracle, it's e2e tests on staging — plus a bot that triages their failures so humans don't have to.

I'm that QA engineer. Here's how we at pdf.net got a GitHub Action + Claude to triage every red run: the architecture, the code, and the rakes we stepped on — the bot blaming innocent people, losing tickets, and staying silent when silence was the worst option. Headline number up front: the whole thing costs less per month than one hour of engineering time.

How we got here

Our team is AI-first by design: developers ship with AI assistants, and instead of scaling manual testing along with headcount, we bet on automation. Autotests aren't a safety net "just in case" — they're the primary trigger that we're doing something wrong.

We release once a day, and originally the release was where everything got verified: fifteen-plus PRs roll up, tests run, and if something's red, the one QA engineer investigates. A single release could contain several broken PRs at once, each to be identified among 17+ suspects while the release waits. Mornings turned into detective work.

So we shifted the loop left. Quick infra note: every PR gets a Vercel preview (a dev environment — running the full e2e scope there makes no sense: it's slow, and half the features sit behind flags that don't match production), while staging is effectively pre-prod: every merge lands there, and code ships to production from there. E2e now runs on every merge to staging — the suspect range becomes one fresh merge instead of a day's batch, and the release turns into a final "works assembled" check plus manual testing that automation can't cover.

The price: red runs now happen potentially after every one of those 17 merges, each needs a quick investigation, and until it's done the release is in question. Hiring a person for "look at a failed run and decide who to hand it to" is overkill — that's not a role, it's a distraction. The reasoning wrote itself: if our tests are automation, triaging their failures should be automation too.

Handily, the failure sources are a closed list — a PR regression in the main repo, the neighboring backend repo (its deploys trigger our e2e too), external services, CI infrastructure, or a flake. Manual triage was always the same mechanical chain over structured inputs: failed tests, diffs, logs → a three-way call. Exactly the kind of task LLMs are good at.

Architecture: verdict → routing

The triage is a reusable composite GitHub Action, invoked after every e2e run — red and green. Around it: Allure TestOps (test results), Linear (tickets), Slack (notifications), Claude API (the brain).

                      e2e run on staging
                             │
               red ──────────┴────────── green
                │                          │
           AI verdict                 Recovery: close tickets,
                │                     rebase-nudge PR authors
     ┌──────────┴───────────────┐
likely-pr-related      likely-not-pr-related
     │                 / insufficient-data
Linear ticket                   │
+ Slack thread         Slack heads-up + ONE auto-rerun
                                │
                     ┌──────────┴──────────┐
                rerun green            rerun red
                     │                      │
              it was a flake —      escalation: ticket
              note it, move on      + @oncall mention

On a red run the bot collects failed tests with steps from Allure, job logs, the deploy commit range, and the diffs of the PRs that went in — and asks Claude (Opus 4.8, Sonnet 5 fallback) for a verdict plus a fix hypothesis:

const verdict = await askClaude({ failedTests, jobLogs, deployRange, prDiffs })
// => 'likely-pr-related' | 'likely-not-pr-related' | 'insufficient-data'

if (attempt === 1 && verdict !== 'likely-pr-related') {
  await slack.post(headsUp(verdict))  // "doesn't look PR-related — rerunning"
  return requestRerun()               // exactly one auto-rerun
}

// PR-related, or a red rerun → Linear ticket + Slack post,
// on a red rerun — with an on-call mention
await createLinearIssue({ verdict, hypothesis, failedTests })

The key decision: the verdict picks a route, it doesn't just write text. PR-related → ticket with a hypothesis and a Slack thread. Anything else → no ticket yet: a heads-up and one auto-rerun. Green rerun — it was a flake, no noise. Red rerun — ticket plus an on-call mention.

🔴 e2e failed on staging — 3 tests
Verdict: likely-pr-related
Hypothesis: regression in PR #1234 — the file upload handler
changed, tests fail on the "attach document" step
Ticket: QA-482 · Staging: 🔴 red — release on hold

Tickets land in one Linear queue, with labels telling frontend deploys from backend ones. One manual habit pays for it all: a confirmed real bug gets a bug label — so "how many bugs did autotests catch, and how much was noise" is a plain queue filter.

Noise control, four mechanisms: dedup (repeat failures match open auto-tickets instead of spawning new ones), auto-close (green run closes the ticket, posts recovery with outage duration, and nudges open PRs to rebase), two channels (ticket-backed posts go to the important channel, rerun chatter stays in the working one), and manual muting (tickets closed without a fix go to Canceled, not Done — recovery leaves them alone).

Neat on paper. Now the four rakes.

Rake #1: the ticket stuck open forever

Recovery writes to three places: a Slack post, the Done transition, and a recovered-at: <sha> marker comment so the next green run skips the ticket. The first version wrote the marker first. Slack post succeeds, Done transition fails on a network hiccup — the marker is already there, and every following run says "already processed". Forever.

The fix is a classic idempotent sequence — each step guarded by its own marker, "fully processed" written last:

if (!hasSlackPostedComment(issue.comments)) {
  const resp = await slack.postMessage(buildRecovery({ issue, attribution, cause }))
  await linear.addComment(issue.id, `slack-recovery-posted:${resp.ts}`)
}
if (issue.stateType !== 'completed') {
  await linear.transitionToDone(issue.id)
}
if (!hasRecoveredAtComment(issue.comments)) {
  await linear.addComment(issue.id, `recovered-at:${sha7}`)  // strictly last
}

Fail in the middle — the next run completes the rest without duplicating anything. Bonus: state lives in the ticket's own comments — no external database, human-readable, garbage-collected with the ticket.

Rake #2: the bot that blamed the innocent

Recovery posts ended with "thanks for the fix, @author" — author computed from the attached fix-PR, or, as a fallback, from the commits in the green run's deploy range. But when a ticket closed because an external service came back to life, the fallback still found an "author": some random person whose innocent PR rode the same deploy. Publicly thanked for a fix that never existed. A couple of those and trust in the bot is gone.

The fix is a recovery-cause classifier — a pure function, no AI:

type RecoveryCause = 'code-fix' | 'external' | 'ambiguous'

const classifyFixCause = ({ deployRange, originalVerdict }): RecoveryCause => {
  const codeFiles = deployRange.files.filter(f => isCodeFile(f.filename))
  if (codeFiles.length === 0) return 'external'       // only docs/lockfiles deployed
  if (originalVerdict === 'likely-not-pr-related') return 'ambiguous'
  return 'code-fix'
}

Attribution now fires only for code-fix; external and ambiguous use a no-names template. ambiguous doubles as telemetry: how often the model's verdict diverged from the actual cause.

Rake #3: mixed runs and the false "rerun helped"

The first dispatcher was if (hasFailures) runAnalysis() else runRecovery(). Breaks on a mixed run: old ticket's tests are fixed, but something new failed — the run is red, recovery never fires, the fixed ticket hangs open. The fix: both paths run on every invocation, recovery first, each no-oping when idle:

await runRecovery(ctx)   // close everything that recovered
await runAnalysis(ctx)   // triage everything that failed

Immediate blast radius: our "rerun helped" signal was just runAttempt >= 2, which was safe while recovery ran only on green runs. Once it ran always, the bot cheerfully posted "✅ recovered — auto-rerun helped" on a red rerun. Hence the explicit guard: runAttempt >= 2 && failureSet.size === 0. Every check that implicitly relied on the old call order has to become explicit.

Rake #4: the run triggered by someone else's deploy

Our e2e lives in the frontend repo and walks the full user scenario through the UI — so it catches backend breakage too, and the backend's staging deploy triggers the same workflow via workflow_dispatch. The GitHub Actions quirk you learn the hard way: a dispatched run has no push context. GITHUB_EVENT_BEFORE is empty, there is no deploy range — and the bot died silently: no ticket, no post. Red staging, and silence. The worst failure mode for a triage system: everyone assumes quiet bot = all under control.

The fix — the deploy range became an explicit input:

- uses: our-org/staging-triage@v1.6
  with:
    deploy-repo: our-org/backend        # whose deploy triggered the e2e
    deploy-before-sha: ${{ inputs.before-sha }}
    deploy-head-sha: ${{ inputs.head-sha }}

Everything downstream — diffs for the verdict, authors for attribution and mentions — now comes from the repo that actually deployed. One triage, both repos.

How we tested it

You can't unit-test a system whose behavior lives across CI, Linear, Slack, and Allure. Two things worked: a smoke PR walked commit by commit through the whole state series — first failure (ticket created?), repeat failure (dedup quiet?), mixed run (both paths fire?), green run (closed + recovery post?) — real runs, real APIs, no mocks. And a kill switch: manual workflow_dispatch is suppressed unless STAGING_TRIAGE_SIMULATED=true, so you can poke the workflow without spamming Linear.

What the bot can't do

The most dangerous failure mode of an LLM classifier isn't a wrong verdict — it's confident text about nothing: "possibly a flake", "check the environment", "probably a cold start". So the prompt is built on bans, not wishes:

Verdict first, justified in one sentence with a concrete signal (a diff file, a trace quote, a locator). The hypothesis format is keyed to the verdict: likely-pr-related → file:line and what to change; likely-not-pr-related → facts with quotes, what's missing, where a human should look.
Hard bans: no files or locators not present in the provided context; no generic phrasing without a quote; and insufficient-data is declared a correct, useful answer — otherwise the model invents hypotheses for the sake of pretty text.
Budgeted context: traces, logs, commit range, full PR diffs and affected files under a 1.5 MB budget; on overflow files drop first, diffs trim from the tail, and a truncation note tells the model not to conclude from missing data.

What still happens: hypotheses right about the area, wrong in the details — the first critical bug the bot caught, it pointed at the right place, but a human wrote the actual fix, just faster. That's fine. The bot doesn't replace an engineer — it replaces the first 15–30 minutes of their work.

Numbers and gut feel

Honest part first: we haven't collected rigorous metrics, and two months is a small sample. But releases got noticeably faster — bugs are localized when the code lands on staging, not on release morning, and the 17-suspects detective game is gone. The bug label groundwork means in six months there'll be real numbers.

Cost: one model call per red run (recovery needs no AI), typically 30–80K input tokens — $0.3–0.5 per red run at Opus 4.8 prices, ~$2.5 worst case, $30–70 a month total. The QA team has grown since — I went from the only QA to leading it — but triage stayed with the bot: people do automation and agent experiments, not red-run duty. The automation was born from a shortage of hands and stayed because it's simply better.

"But off-the-shelf tools exist!"

They do. Cognition sells Devin Auto-Triage; GitHub shipped a “Fix with Copilot” button on failed workflow runs. We saw all that and deliberately built our own — because our task isn't "fix the red CI". It's a decision about the fate of a release: classify the cause, pick a route (ticket / rerun / escalation), dedup, auto-close, wake nobody without a reason — welded to our exact stack (Allure TestOps, Linear, Slack, two repos) and our processes.

And the price tag: for an AI-first team this is days, not weeks — the code is almost entirely written with Claude Code. The honest division of labor: the human part is the prototype and the architecture — verdict separated from routing, idempotent recovery steps, pure-function classifiers, state in ticket comments instead of a database. AI writes great code inside a given structure and speeds iteration up by an order of magnitude; the structure that survives six releases in two months is set by a human.

Why not open-source it, then? Similar "AI triages your failed CI" actions sit at 3–11 GitHub stars, and ours needs the exact Linear + Slack + Allure TestOps + Anthropic combination — that audience is vanishingly small, while baseline "AI explains the failure" is arriving in GitHub out of the box. What's worth open-sourcing here are the ideas, not the code — which is exactly what this article is.

AI-first means AI-first.

Selective test execution mechanism with Playwright using GitHub Actions

Denis Skvortsov — Sat, 24 May 2025 09:49:04 +0000

TL;DR

Selective test execution: run only the tests related to the actual code changes.

Saves time and resources: speeds up the process and reduces CI/CD load, especially in cloud environments.

Works for both monorepos and split repositories: the solution fits projects with either a monorepo or separate frontend/backend repositories.

Uses GitHub Actions and Playwright: configures CI/CD to filter tests by tags and run only the relevant ones.

Example implementation: available in a public GitHub repository

Introduction

When I first faced the task of setting up testing, I realized that running the entire test suite after every code change is not just slow - it's wasteful. Especially in large projects where microservices, frontend, and backend each have their own test sets. In such setups, running all tests for every commit or PR is unnecessary and significantly slows down the development cycle.

For example, if a frontend developer changes a button or a small UI component, why would we run all backend tests? Or if a backend developer updates a single endpoint, there’s no point in triggering all UI tests - they’re completely unrelated. In those cases, tests become unnecessary overhead that slows down progress.

In some teams, test success is a hard requirement before merging code. If you can’t merge until all tests pass - but your change affects only a small part of the system - triggering all tests becomes a bottleneck.

To solve this, I implemented selective test execution - running only the tests that are actually affected by code changes. This approach helps save both time and infrastructure resources, making the testing and release process faster. In this article, I’ll share my experience and show how to set up such a mechanism using Playwright and GitHub Actions - whether you’re working in a monorepo or with separate frontend/backend repositories.

What problem are we solving?

Running the full e2e test suite on every code change is not always justified. In projects that include frontend, backend and shared modules, this often leads to problems like:

Developers waiting for CI feedback longer than it takes to write the fix itself
CI infrastructure usage skyrockets (and if you’re in the cloud - so do the costs)
Tests are triggered that have nothing to do with the actual changes

Typical scenarios:

A frontend developer changes a visual component, but all backend test chains are executed
A backend developer tweaks business logic, and the entire UI test suite runs too
A shared utility is updated, and suddenly no one knows whether to run a full regression or not

It gets even worse when tests are a mandatory check before merging. Even a minor change can block the entire process - while dozens or even hundreds of unrelated tests are waiting to complete.

My goal wasn’t just to speed up testing. I wanted CI to run only what truly matters. Automatically. Without manual rules or exception lists.

How i implemented it

To demonstrate how you can run only the relevant e2e tests, I put together a small monorepository with a simple but flexible architecture. This isn’t a production-ready setup - it’s a demo project, intentionally simplified to make the mechanism easy to understand and adapt to any real-world structure.

My main goals were:

Everything should work transparently - no magic config files
The implementation should be reusable - something you can easily apply to another project
The structure should remain flexible - easy to extend with new services without rewriting the pipeline

Project structure:

├── frontend/                    # Frontend applications
│   └── apps/                    # Frontend microservices
│       ├── microservice1/
│       └── microservice2/
│       └── shared/             # Common frontend components

├── backend/                     # Backend services
│   └── apps/                    # Backend microservices
│       ├── microservice3/
│       ├── microservice4/
│       └── microservice5/

├── .github/                     # GitHub Actions
│   ├── workflows/               # CI/CD configuration
│   │   └── e2e-runner.yml       # Selective test runner
│   └── preconditions/           # Reusable actions
│       └── e2e/                 # E2E tests environment setup
│           └── action.yml       # Composite action for environment setup

└── tests/                       # Tests
    └── e2e/                     # E2E tests

Everything is split into logical areas - just like in real projects. There's frontend, backend, a shared area and a dedicated folder for e2e tests. The entire pipeline runs on GitHub Actions. Dependencies are installed using a reusable precondition action. The core logic is tag-based.

The tests themselves are intentionally primitive - because this article isn’t about test coverage or scenarios, it’s about the mechanism for running only what matters. Everything else is just context.

1. Tagging Tests

To determine which tests should run, I use tags directly in the test definitions. The logic is simple: if a PR changes apps/microservice3, the CI system looks for e2e tests tagged with @apps/microservice3 and runs only those.

Each tag is tied to a specific microservice or module. For example:

  test('Test 3', { tag: '@apps/microservice3' }, async ({ ui }) => {
    await ui.google.goto();
    await ui.google.openAppsMenu();
    await ui.google.assertServiceVisible('Maps');
    await ui.google.assertServiceVisible('Gmail');
  });

If microservice3 is affected, this test will be included. If another service is changed, it will be skipped-even if it's in the same file.

I chose the format @apps/<service-name> for two reasons:

It matches the actual project structure.
It’s easy to extract from file paths using grep and sed.

Another example:

  test('Test 1', { tag: '@apps/microservice1' }, async ({ ui }) => {
    await ui.google.goto();
    await ui.google.openAppsMenu();
    await ui.google.assertServiceVisible('YouTube');
    await ui.google.assertServiceVisible('YouTube Music');
  });

If a test is not tagged, it won’t be picked up during selective test runs. For example:

test('Test 5', async ({ ui }) => {
  await ui.google.goto();
  await ui.google.openAppsMenu();
  await ui.google.assertServiceVisible('Calendar');
});

This test will only run during a full test execution (for example, when shared or tests/e2e changes are detected).

Tagging is a core part of this mechanism. It doesn’t require additional tooling and can be maintained manually if needed.

Playwright docs on tags: Test annotations

2. Analyzing сhanges and mapping to tags

The next step is to determine what exactly has changed in the PR and which tests should be triggered. This is done in GitHub Actions using a simple git diff.

What we look for:

Changes in paths like apps/microserviceX → means a specific service was affected
Changes in shared → means potentially all services are affected
Changes in tests/e2e → likely the tests themselves were modified, so the full suite should be executed

Here’s a simplified example from the Find changes step:

changed_apps=$(git diff --name-only origin/main HEAD | grep -E "/apps/[^/]+/" || true)
changed_test_files=$(git diff --name-only origin/main HEAD | grep -E "^tests/e2e/" || true)

Then we normalize the paths. For example, frontend/apps/microservice1/pages/page.tsx becomes apps/microservice1, which maps directly to the tag @apps/microservice1.

full_paths=$(echo "$changed_apps" | sed -E 's#^(.*/apps/[^/]+)/.*#\1#' | sort -u | paste -sd "|" -)
test_paths=$(echo "$full_paths" | tr '|' '\n' | sed -E 's#.*(apps/[^/]+)#\1#' | paste -sd "|" -)

Then what happens:

If the changes include shared or tests/e2e, we skip filtering and run all tests
If only specific services were changed, we convert the paths into tags, which are passed to Playwright via --grep

The result is saved into a GitHub Actions output variable called test_scope, which is used in later steps:

echo "test_scope=$test_paths" >> $GITHUB_OUTPUT

If test_scope ends up empty, it means either:

No relevant changes were found
Or no tests are tagged to match the changes

In such cases, you can either skip test execution or fall back to running the full suite - depending on your project’s policy.

3. Running tests with `grep`

Once we’ve determined which parts of the code have changed and built a list of tags, the next step is simply passing those tags to Playwright. For that, we use the built-in --grep flag, which filters the tests by tags.

Example run:

If the PR affects both apps/microservice1 and apps/microservice4, the resulting scope looks like this:

test_scope=apps/microservice1|apps/microservice4

Then the tests are run like this:

npx playwright test --grep "@apps/microservice1|@apps/microservice4" || true

Why || true?

Because grep might return no matches - for example:

The service is new and doesn’t have tests yet
The change is minor and not yet covered
Tests exist but lack proper tags

In these cases, we don’t want the CI to fail. It’s okay to skip test execution if no relevant tests were found. Failure should only happen when tests exist and they fail - not when there are simply no tests to run.

When do we run all tests?

If the changes include any of the following:

A shared directory (frontend or backend)
Files in tests/e2e
Or if test_scope is completely empty (depending on your policy)

We simply skip filtering and run the full suite:

if: needs.detect-changes.outputs.test_scope == ''

And then:

npx playwright test

All of this is wrapped inside a pipeline with two jobs: selective-tests and all-tests. Based on the content of test_scope, only one of them runs.

What we get in the end

After implementing selective test execution on GitHub Actions, the benefits were immediately obvious - both in terms of human effort and infrastructure usage.

Faster builds
CI no longer runs the entire test suite for every little change. If only one microservice is affected, only its e2e tests are triggered. As a result:

Tests run significantly faster
You get feedback almost immediately
Releases are no longer blocked by irrelevant checks

Resource savings
If you're running tests in the cloud, CI/CD can get expensive - especially when all tests are triggered for every pull request. Selective execution helps save actual money by avoiding unnecessary resource usage.

Easy to maintain
You don’t need to manually build complex test mappings. The entire system is driven by:

A simple tag in each test (@apps/xxx)
git diff and bash (2 lines of code)
grep

The approach is flexible and adaptable to any structure. You can filter by modules, features, user flows, folders, roles - whatever makes sense in your project.

Works out of the box
This mechanism performs well in:

Monorepos
Projects with split frontend and backend
Any project where e2e tests are a mandatory merge requirement

Want real-world scenarios?
You can explore common situations - shared modules, new services, test changes, combinations - in the project’s README

Example GitHub Actions workflow

name: E2E Tests
on:
  pull_request:
    branches: [ main ]

jobs:
  detect-changes:
    runs-on: ubuntu-latest
    outputs:
      test_scope: ${{ steps.scope.outputs.test_scope }}
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Find changes
        id: changes
        run: |
          echo "🔄  Analyzing changes in PR..."
          changed_apps=$(git diff --name-only origin/main HEAD | grep -E "/apps/[^/]+/" || true)
          changed_test_files=$(git diff --name-only origin/main HEAD | grep -E "^tests/e2e/" || true)

          echo "📦 Files changed:"
          [ -z "$changed_apps" ] && echo "  No changes in files" || echo "$changed_apps" | sed 's/^/  /'

          echo "🧪 E2E tests:"
          [ -z "$changed_test_files" ] && echo "  No changes in e2e tests" || echo "$changed_test_files" | sed 's/^/  /'

          echo "✨ Affected services:"
          full_paths=$(echo "$changed_apps" | sed -E 's#^(.*/apps/[^/]+)/.*#\1#' | sort -u | paste -sd "|" -)
          [ -z "$full_paths" ] && echo " No changes in services" || echo "$full_paths" | tr '|' '\n' | sed 's/^/  /'

          test_paths=$(echo "$full_paths" | tr '|' '\n' | sed -E 's#.*(apps/[^/]+)#\1#' | paste -sd "|" -)
          test_files=$(echo "$changed_test_files" | paste -sd "|" -)

          echo "test_paths=${test_paths}" >> $GITHUB_OUTPUT
          echo "changed_test=${test_files}" >> $GITHUB_OUTPUT

      - name: Check shared modules and modified e2e tests
        id: shared
        run: |
          test_paths="${{ steps.changes.outputs.test_paths }}"
          changed_test="${{ steps.changes.outputs.changed_test }}"

          echo "🔄 Checking shared modules and e2e tests:"

          has_shared=$(echo "$test_paths" | tr '|' '\n' | grep -q "shared" && echo "true" || echo "false")
          has_e2e_changes=$([ ! -z "$changed_test" ] && echo "true" || echo "false")

          $has_shared && echo "⚠️ Changes in shared modules detected" && echo "$test_paths" | tr '|' '\n' | grep "shared" | sed 's/^/  /'
          $has_e2e_changes && echo "⚠️ Changes in e2e tests detected" && echo "$changed_test" | tr '|' '\n' | sed 's/^/  /'

          { $has_shared || $has_e2e_changes; } && test_paths="" || echo "✅ No changes in shared modules or e2e tests"

          echo "test_paths=$test_paths" >> $GITHUB_OUTPUT

      - name: Set final scope
        id: scope
        run: |
          test_paths="${{ steps.shared.outputs.test_paths }}"
          echo "🔄 Result:"
          [ -z "$test_paths" ] && echo " ✅ All tests will be run" || {
            echo " ✅ Running tests for:"
            echo "$test_paths" | tr '|' '\n' | sed 's/^/    /'
          }
          echo "test_scope=$test_paths" >> $GITHUB_OUTPUT

  selective-tests:
    needs: detect-changes
    if: needs.detect-changes.outputs.test_scope != ''
    timeout-minutes: 60
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.52.0
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/preconditions/e2e
      - name: Run selective tests
        run: npx playwright test --grep "${{ needs.detect-changes.outputs.test_scope }}" || true

  all-tests:
    needs: detect-changes
    if: needs.detect-changes.outputs.test_scope == ''
    timeout-minutes: 60
    runs-on: ubuntu-latest
    container:
      image: mcr.microsoft.com/playwright:v1.52.0
    steps:
      - uses: actions/checkout@v4
      - uses: ./.github/preconditions/e2e
      - name: Run all tests
        run: npx playwright test

Can you run only the tests that were changed?

This is a common question that usually comes up first:
"If we already know which files have changed - why not run only the tests that were also modified?"

At first glance, it sounds reasonable. But in practice it’s not always a good idea.

Why I didn’t go down that path

In my setup, I run all tests related to the changed components, not just the .spec.ts files that were edited in the PR. The reason is simple - in some projects, tests are not properly isolated.

A typical example:

one test modifies some data,
another test relies on the state left behind by the first one,
or there’s a shared setUp that affects the behavior of all tests.

This is an anti-pattern, of course, but it's still common - especially in older or fast-growing projects.

If you run only the modified test files, you can easily end up with a green build - even though the logic is actually broken.
That’s why I went with a safer approach:
we filter by the affected areas, but within that area, we run all the tests - even if the test files themselves weren't changed.

What about `--only-changed`?

Playwright does support a --only-changed flag that runs only the .spec.ts files which were changed.
It can be helpful as a temporary solution or for small PRs.

But it’s important to understand: this flag only works at the file level.
It doesn’t track which modules or helpers were changed, nor does it understand which tests depend on them.

So if you modify something like auth.ts, which is used across all tests - --only-changed won’t pick that up, because the test files themselves didn’t change.

What you can do if you want to take it further

If your tests are well-isolated and your project architecture is clean, it’s possible to run only those tests that are truly affected by a change.

At the git diff stage, you can track not only changes in app code, but also updates to shared modules or e2e utilities. To identify which tests depend on those changes, you can build a dependency graph between source files and test files - using tools like ts-morph or dependency-cruiser. This graph reveals which .spec.ts files import or transitively rely on the modified code.

This approach works best when your test structure is modular and the dependency graph is accurate and regularly maintained. Without that, the risk of silently skipping important tests increases.

That’s why in many cases, it's safer and more predictable to run all tests within the affected scope - even if only a small change was made.
It keeps things simple and reduces the chance of hidden regressions in CI.

Conclusion

Selective execution of e2e tests isn’t a silver bullet - but it’s an effective way to reduce build time, lower CI load, and speed up the development cycle. It’s especially valuable in projects where tests are a required condition for merging, and every minute of CI time matters.

All you need is to:

tag your tests appropriately
analyze changes in the PR
and run tests using the --grep filter

The solution is simple, flexible and reusable. You can apply it in monorepos, in projects with separate frontend and backend repos or even as a shared internal standard within your team.

If you want to take this further - like running only the truly modified tests - that’s also possible if architecture makes it achievable.

If you’ve solved a similar problem differently - I’d love to hear how you approached it. Feel free to share your solutions.

Practical use of Cursor and MCP Playwright in test automation

Denis Skvortsov — Wed, 23 Apr 2025 11:12:02 +0000

Introduction
This article is not a documentation review. It's my personal experience working with the Cursor and Playwright MCP tools for frontend test automation (JavaScript/TypeScript). I want to share how these tools truly help in everyday work, especially when it comes to writing or improving automated tests.
The topic of automation is actively developing, and, perhaps in the near future, we will be able to automate tests almost by sheer willpower. But for now, I'll share what I've already found. Many developers are actively using AI for code generation, such as GitHub Copilot, but for me, Cursor in combination with Playwright MCP provides much more context. While these are far from perfect tools-sometimes they fix unnecessary places or don't work exactly as expected-they are still a powerful addition to the process.
In this article, I will show how to use Cursor and Playwright MCP in practice, with test examples and explanations of where and how these tools help, as well as situations where they are better avoided.
Here are the key points I would like to highlight in this article that may be useful when working with Cursor and MCP Playwright:
Cursor can automatically add data-testid or getByRole to the code using a screenshot of the screen and highlighting the desired area where it should be inserted. This is especially useful for beginner automation engineers.
MCP Playwright is useful for labeled pages, where elements have accessible attributes (e.g., data-testid, aria-label, etc.).
MCP Playwright is ideal for writing simple scripts that do not require complex preconditions and works well for basic E2E testing.
If you're using Cursor and MCP Playwright together, the best solution is to organize a monorepository, where both the frontend and automated tests are located in one repository.

Additionally, it's important to remember that while these tools speed up the process, they do not replace engineering work, especially for more complex scenarios and integration tests.

Test Architecture: Basic principles

Before implementing MCP Playwright, it's important to revisit your testing pyramid and define what e2e tests mean in your context. The testing pyramid can vary for different teams and projects. E2E and integration tests can overlap, but MCP Playwright is useful only for simple scenarios.

My tests and examples will be based on the website google.com, and I will use TypeScript for writing the tests. Google provides good accessibility for element recognition through MCP Playwright, which allows easy creation of stable tests for basic operations, such as searching on the page. I will also use the test architecture from the article Simple and Effective E2E Test Architecture for Playwright.

How MCP Playwright and Cursor fit different types of tests:
E2E Tests: MCP Playwright is ideal for simple e2e tests where you need to check UI interactions, especially if the page is properly labeled. For example, checking a button or an input field. Here, MCP automatically generates code with the correct locators, making test writing easier. However, it's important to remember that MCP Playwright works best with simple scenarios.
Integration Tests: For integration tests where you need to interact with both the UI and the API, traditional automation methods are better. Cursor can help with generating templates and structuring mocks, but the logic of interacting between multiple modules and handling errors requires an engineering approach.
Unit Tests: Unit tests remain the responsibility of developers, but for automation engineers, Cursor can be a useful tool for speeding up test creation, especially for typical cases.

Example of using MCP Playwright in e2e tests
Open Google
Search for "Playwright MCP"
Verify that the search results contain the text "Model Context Protocol"
Open Google again
Search for "Playwright automation"
Verify that the search results contain the word "testing"

This test interacts solely with the UI. For such simple scenarios, MCP Playwright will generate the necessary code, ensuring stable and fast test execution.

Integration Tests: An AQA Engineer's perspective
When it comes to more complex integration tests, Cursor can provide a test template and a structure for mocks, but for complex logic, such as error handling or interaction between multiple modules, you need to apply engineering thinking.
Example
Cursor generated a test for a successful scenario, but it didn't account for all the negative cases related to errors. I had to refine it manually, adding error handling:

This is an example of how Cursor helps with templates, but you still need to tweak it for more complex scenarios. Also, Cursor tends to overcomplicate solutions, so it's important to monitor what it generates and double-check its work.
Cursor works well in monorepositories and can reuse existing UI and API methods. To limit the complexity of the generated code, you can use Cursor rules. Cursor Rules Documentation.
Of course, this is not a solution to all problems, but it can significantly simplify your work.

Unit tests: How Cursor helps speed up test creation
Unit tests are primarily the responsibility of developers, but as an automation engineer, you can also get involved, especially if you actively participate in writing code and understand its logic. Cursor helps generate tests quickly, saving time, especially for common cases.
Example from Practice
With Cursor, I generated several tests for functions like formatCurrency and parseQuery. Here's an example of how it might look:

Cursor suggested edge cases like null, NaN, and empty strings, which are often forgotten in tests. This saves time, eliminates human errors, and helps make tests more comprehensive.

How to use Cursor to add data-testid or modify buttons on the frontend

Cursor has a great feature for adding a screenshot of the page to a request, which is especially useful if you have a monorepository for both frontend and e2e tests. This is a great opportunity to immediately fix a component and label it in an accessible way while writing a script. Of course after making changes, you should verify the display, but this significantly simplifies the process for beginner automation engineers or those just getting familiar with the frontend.

Monorepositories - A great solution for these tools

Although I haven't had the opportunity to work with these tools in a single monorepository where both frontend and backend are located, in practice, when all the automated tests are in the same repository as the frontend, this solution has proven to be excellent. Firstly, you can immediately label the necessary elements on the frontend, and with Cursor, you can quickly pull the required API request or examine how the frontend is structured overall.
Using a monorepository in combination with these tools gives you flexibility and improves collaboration between teams, especially in terms of writing and maintaining tests.

When to use MCP Playwright sensibly
MCP Playwright is really good for testing interfaces when the page is properly labeled, and the scenarios are simple. It speeds up testing and eliminates the need to manually write locators.

Use MCP Playwright if:
The UI is properly labeled with accessibility attributes.
The page being tested is simple: a form, a button, or a basic navigation.
The test logic is as simple as possible.

Don't Use MCP Playwright if:
The interface is custom, and the elements do not have accessible attributes.
The test logic is complex, involving multiple interactions or APIs.
You need to check internal values, not just the visual state.

Conclusion
MCP Playwright and Cursor are powerful tools that can significantly speed up test creation and help with routine tasks. However, they cannot replace the work of an engineer, especially in more complex scenarios.
Cursor is excellent for generating test templates and working with common cases. It helps quickly set up the structure of tests but requires refinement for more complex scenarios, such as error handling or complex logic.
MCP works perfectly when the page is properly labeled with accessibility attributes. It eliminates the need to manually write locators, speeding up the testing process. But if the elements on the page are dynamic or not properly labeled, MCP may not be as reliable, and you should consider using other methods for element search.

In any case, while these tools significantly speed up testing, they do not replace full engineering work. It's important to remember that for complex scenarios or integration tests, despite the convenience of these tools, you will still need manual intervention for correct logic setup.
If you have already used MCP and Cursor in your projects, I'd be interested to hear how these tools helped you in real-world cases.