DEV Community: Gabriel Holmes

Your E2E tests live in a separate repo. That's why nobody sees the breakage coming.

Gabriel Holmes — Fri, 03 Jul 2026 13:57:00 +0000

Many QA teams keep end-to-end automation in a dedicated repo — separate from the app it tests. It's good hygiene: different owners, different release cadence, no test churn polluting app history.

It also quietly removes the one signal that would tell a developer their change broke a test.

The boundary problem

When tests and app share a repo, at least the failing spec shows up in the same CI run and the same blame history. Split them, and the connection vanishes:

The frontend dev opens a PR in the app repo. CI there is green — there are no E2E tests to run.
The PR merges. A scheduled or downstream run in the automation repo goes red hours later.
The QA engineer who owns that repo inherits a failure with no link back to the PR that caused it.

The dev never sees it. The reviewer never sees it. The cost lands on whoever owns the other repo, after the fact. If you're the automation owner, you know this loop: what changed vs what broke vs which specs do I update.

Why normal tooling can't help here

Coverage tools, "affected test" selectors, and CI test-splitting all operate within a single repo and a single test run. They have no concept of "a change here breaks a test there." GitHub's own PR view shows the diff and that repo's checks — nothing about a sibling repo.

The boundary that makes the split clean is the same boundary the tooling can't cross.

What actually connects them: text

Here's the useful observation: your E2E specs are coupled to the app through a small set of textual anchors — data-testid / data-cy attributes, getByRole/getByText targets, aria-labels, element ids, route strings, visible copy.

A PR that renames data-testid="login-btn" or moves /dashboard to /home endangers every spec that references those strings — regardless of which repo the spec lives in. Text doesn't care about repo boundaries. So the check is:

Extract the anchors a PR's diff touches (added/removed lines only).
Search the automation repo's spec files for those anchors.
Anything that matches is a break candidate — surfaced on the PR, at review time.

That's cheap, static, and crosses the boundary that execution-based tooling can't.

Automating it

I'm a QA automation engineer and I built this into a GitHub App — Testward. You add a one-line .testward.yml to the app repo listing your automation repo(s); on every PR it extracts the anchors, scans the linked repos' specs (code search + a tree-grep fallback for large suites), runs an LLM pass to filter candidates that don't actually break, and posts one comment:

🚨 Automation this PR may break:
- acme/web-e2e → tests/login.spec.ts — clicks [data-testid=login-btn]
  (renamed to signin-btn) and asserts the /dashboard redirect (changed to /home)

The dev sees the consequence while they still have the context — and either fixes the selector or pings the automation owner before the merge, not after the red.

Being upfront about the limits: the matching is textual, so dynamic selectors (`row-${id}`) won't match — that's the known gap. And if you want to poke at the anchor-extraction idea without installing anything, there's a free in-browser diff scanner that runs entirely client-side.

Takeaway

If your automation lives in its own repo, no amount of in-repo tooling will warn the app team they're about to break it. The dependency is textual, so the check can be too — and it belongs at review time, where the fix costs a comment instead of an afternoon.

If your team runs a separate automation repo, I'd genuinely like to hear how you handle this today — scheduled runs and hope? A sync meeting? Something smarter?

How to know which tests a PR will break — before CI runs

Gabriel Holmes — Fri, 03 Jul 2026 05:00:53 +0000

Every team has lived this: a pull request looks clean, gets approved, merges — and twenty minutes later the test suite is red. The change was in src/. The failure is in tests/. And nothing in the review surfaced the connection.

Why code review misses it

Code review shows you the diff. It does not show you what depends on the diff. A reviewer reading a one-line selector rename has no way to know that three end-to-end specs select on that exact attribute — especially if those specs live in a folder they never open, or a repo they don't have checked out.

So the breakage is discovered by the most expensive possible detector: a full CI run, after merge, by whoever is on call for the red build. The person with the most context — the author, mid-review — never saw it coming.

The three ways teams try to close the gap

1. Run the whole suite on every PR. Correct, but slow and expensive, and it tells you after the fact — you still wait for the red. On large suites it's minutes-to-hours of feedback latency per push.

2. Coverage-based test selection. Map code to the tests that execute it and run only those. Powerful for unit tests, but it needs instrumentation, breaks down for E2E (where the "coverage" is a running browser), and says nothing across repo boundaries.

3. Static impact analysis at review time. Read the diff, extract what it changes that tests depend on — selectors, routes, ids, visible text — and match those against the test files. This is cheap, needs no instrumentation, and runs the moment the PR opens.

What "what tests depend on" actually means

End-to-end and integration tests are coupled to the app through a small, identifiable set of anchors:

Selectors / test ids — data-testid, data-cy, getByRole, CSS, XPath
Routes — URL literals the test navigates to or asserts on
Visible text & labels — button copy, headings, aria-labels the test queries
Element ids and names — the locators page objects hang off

If a PR changes one of these, any test referencing it is a break candidate. That's a tractable matching problem you can run in seconds — no test execution required.

Want to see it on a real diff? I built a free scanner that runs this extraction entirely in your browser (nothing gets uploaded): testward.app/diff-scanner

The cross-repo blind spot

Here's where it gets interesting for QA teams specifically. Many of us keep automation in a dedicated repo — separate from the app it tests. Good hygiene, but it severs the only signal: the frontend dev's PR is green (no E2E tests live there), and the automation repo goes red hours later with no link back to the cause.

Coverage tools, affected-test selectors, and CI test-splitting all operate within a single repo and a single test run. None of them can say "a change here breaks a test there."

How I automated this

I'm a QA automation engineer, and this loop ate enough of my afternoons that I built a GitHub App for it — Testward. On every PR it:

Reads the diff and extracts the anchors the change touches.
Scans your test files — same repo, or a separately-linked automation repo (one line of config).
Runs an LLM confirmation pass that names the specific specs likely to break and why.
Posts one sticky comment: risk level, affected specs, reasons.

The reviewer sees the consequence next to the cause, while the author still has the context to fix it — or update the test in the same PR.

Known limitation worth being upfront about: dynamic selectors (`row-${id}`) can't be matched statically — that's the gap I'm working on. If your suite leans on template-literal test ids, the anchor model will miss those.

Takeaway

You don't need to run tests to know a PR endangers them. The dependency between app code and test code is mostly textual — selectors, routes, labels — and you can check it at review time, when the fix is cheapest. Whether you script it yourself or use a tool, moving that signal left is one of the highest-leverage things a QA team can do.

Questions or war stories about PRs silently breaking your suite? I'd genuinely like to hear them — especially from teams whose automation lives in a separate repo.