How to know which tests a PR will break — before CI runs

Gabriel Holmes — Fri, 03 Jul 2026 05:00:53 +0000

Every team has lived this: a pull request looks clean, gets approved, merges — and twenty minutes later the test suite is red. The change was in src/. The failure is in tests/. And nothing in the review surfaced the connection.

Why code review misses it

Code review shows you the diff. It does not show you what depends on the diff. A reviewer reading a one-line selector rename has no way to know that three end-to-end specs select on that exact attribute — especially if those specs live in a folder they never open, or a repo they don't have checked out.

So the breakage is discovered by the most expensive possible detector: a full CI run, after merge, by whoever is on call for the red build. The person with the most context — the author, mid-review — never saw it coming.

The three ways teams try to close the gap

1. Run the whole suite on every PR. Correct, but slow and expensive, and it tells you after the fact — you still wait for the red. On large suites it's minutes-to-hours of feedback latency per push.

2. Coverage-based test selection. Map code to the tests that execute it and run only those. Powerful for unit tests, but it needs instrumentation, breaks down for E2E (where the "coverage" is a running browser), and says nothing across repo boundaries.

3. Static impact analysis at review time. Read the diff, extract what it changes that tests depend on — selectors, routes, ids, visible text — and match those against the test files. This is cheap, needs no instrumentation, and runs the moment the PR opens.

What "what tests depend on" actually means

End-to-end and integration tests are coupled to the app through a small, identifiable set of anchors:

Selectors / test ids — data-testid, data-cy, getByRole, CSS, XPath
Routes — URL literals the test navigates to or asserts on
Visible text & labels — button copy, headings, aria-labels the test queries
Element ids and names — the locators page objects hang off

If a PR changes one of these, any test referencing it is a break candidate. That's a tractable matching problem you can run in seconds — no test execution required.

Want to see it on a real diff? I built a free scanner that runs this extraction entirely in your browser (nothing gets uploaded): testward.app/diff-scanner

The cross-repo blind spot

Here's where it gets interesting for QA teams specifically. Many of us keep automation in a dedicated repo — separate from the app it tests. Good hygiene, but it severs the only signal: the frontend dev's PR is green (no E2E tests live there), and the automation repo goes red hours later with no link back to the cause.

Coverage tools, affected-test selectors, and CI test-splitting all operate within a single repo and a single test run. None of them can say "a change here breaks a test there."

How I automated this

I'm a QA automation engineer, and this loop ate enough of my afternoons that I built a GitHub App for it — Testward. On every PR it:

Reads the diff and extracts the anchors the change touches.
Scans your test files — same repo, or a separately-linked automation repo (one line of config).
Runs an LLM confirmation pass that names the specific specs likely to break and why.
Posts one sticky comment: risk level, affected specs, reasons.

The reviewer sees the consequence next to the cause, while the author still has the context to fix it — or update the test in the same PR.

Known limitation worth being upfront about: dynamic selectors (`row-${id}`) can't be matched statically — that's the gap I'm working on. If your suite leans on template-literal test ids, the anchor model will miss those.

Takeaway

You don't need to run tests to know a PR endangers them. The dependency between app code and test code is mostly textual — selectors, routes, labels — and you can check it at review time, when the fix is cheapest. Whether you script it yourself or use a tool, moving that signal left is one of the highest-leverage things a QA team can do.

Questions or war stories about PRs silently breaking your suite? I'd genuinely like to hear them — especially from teams whose automation lives in a separate repo.