E2E tests can pass while testing nothing

Code-in-the-browser, code-server, had an E2E suite that looked healthy. CI was green. The PR had been reviewed. But one test file still contained this:

it.only(...)

That one line quietly disabled the rest of the tests in the file. Eight tests did not run for seven months, while the check still passed. The fix eventually merged in coder/code-server#7845.

This is not a story about code-server being careless. It is almost the opposite: even mature projects, reviewed PRs, and AI-reviewed changes can end up with E2E tests that are green but do not prove the thing they appear to prove.

I mean "testing nothing" in a narrow sense. The test may execute, and CI may pass, but the assertion does not observe the user-visible condition it claims to protect.

I started collecting these cases while fixing Playwright and Cypress tests across open-source repositories. The same shapes kept appearing.

The same shape in other projects

Carbon had an assertion like this (carbon#22564):

expect(page.locator('.cds--progress-step--complete')).toBeTruthy();

page.locator(...) is not a DOM lookup result. It is a Locator object. The object is truthy whether the element exists or not, so the assertion can pass even when the UI is wrong. The intended check is closer to this:

- expect(page.locator('.cds--progress-step--complete')).toBeTruthy();
+ await expect(page.locator('.cds--progress-step--complete')).toBeVisible();

Ghost had a Playwright web-first assertion without await (Ghost#28712). The line looked like an assertion, but it did not wait for the condition it was supposed to verify.

Strapi had this pattern (strapi#26630):

expect(likeButton.isDisabled()).toBeTruthy();

In Playwright, methods such as isDisabled(), isVisible(), and isHidden() return Promise<boolean>. The Promise object itself is truthy, so the assertion can pass regardless of the actual button state. SvelteKit had the same family of issue in kit#16068.

These were not synthetic examples: the broader pass has produced 12 merged upstream fixes so far.

The details differ, but the underlying failure is the same: "there is a test" and "the test can fail for the intended regression" are different claims.

Checks that catch the low-hanging fruit

These are not a complete review. They are just fast checks that have found real problems in real Playwright/Cypress suites.

# 1. Focused tests accidentally committed
rg "(it|test|describe)\.only\(" e2e tests cypress playwright

# 2. Truthiness checks on Locator-like objects or handles
rg "expect\([^)]*\)\.(toBeTruthy|toBeDefined|not\.toBeNull)\(" e2e tests cypress playwright

# 3. Playwright web-first assertions without await
rg "^[[:space:]]*expect\(.+\)\.to(BeVisible|BeHidden|BeEnabled|HaveText|HaveURL|HaveTitle|HaveCount)\(" e2e tests playwright

# 4. State reads whose result is discarded
rg "^[[:space:]]*(await )?[A-Za-z_.]+\.(isVisible|isEnabled|isDisabled|isHidden)\(\)[[:space:]]*;?[[:space:]]*$" e2e tests cypress playwright

The first two are usually low-noise. The third and fourth need context before you automate fixes. But as a first pass, they are useful because they look for tests that can stay green while asserting little.

Some of this belongs in lint rules. I split out the lintable subset into eslint-plugin-playwright-silent-pass and eslint-plugin-cypress-silent-pass. Grep is good for a quick audit; ESLint is better for preventing repeats.

npm i -D eslint-plugin-playwright-silent-pass   # or eslint-plugin-cypress-silent-pass

What general reviewers tend to miss

At first I assumed general AI PR reviewers would catch most of this. To check that, I re-reviewed 100 already-AI-reviewed open-source E2E PRs across 77 repositories. In the published benchmark, a neutral LLM judge established 110 material E2E-test trust issues in those PRs (results).

The comparison was:

• e2e-reviewer: 78 / 110 material E2E issues found, 0 false positives
• lint (eslint-plugin-playwright / eslint-plugin-cypress): 45 / 110 found, 0 false positives
• General AI PR reviewers' inline spec comments: 10 / 110 found, 72 off-target/noise comments relative to this narrow ground truth

I do not think this means general reviewers are bad. They are trying to review architecture, types, security, performance, style, and product logic at the same time. This problem is narrower. You have to keep asking: can this assertion fail? Does this locator observe the DOM state, or just construct an object? Did an async assertion get dropped on the floor?

A general reviewer can notice some of that. A dedicated E2E trustworthiness pass can make it the whole job.

What I built

e2e-skills is a small set of agent skills for Playwright and Cypress E2E work. The goal is not to ask "does a test exist?" but "can this test actually fail when the product regresses?"

• e2e-reviewer statically reads Playwright/Cypress specs and Page Objects, then checks 24 weak-diagnostic anti-patterns: focused tests, locator truthiness, missing await, discarded state reads, loose status assertions, and related patterns that keep CI green while proving little.
• playwright-test-generator is for creating new Playwright coverage. It first looks for coverage gaps, then explores the live app in a browser, drafts tests for real user flows, pauses at an approval gate, and finally reviews the result with the same weak-assertion rules. It is closer to "find the flow worth proving and write fail-capable assertions" than "dump a test file into the repo."
• playwright-debugger reads Playwright reports, traces, screenshots, and CI output to classify failures: timing, selector drift, auth/session setup, environment trouble, or a real product regression.
• cypress-debugger does the same kind of failure classification for Cypress mochawesome/JUnit output.

The scope is deliberately narrow: Playwright and Cypress only. I would rather catch common fail-open patterns in those two frameworks well than claim shallow support for every browser automation stack.

Repo: github.com/voidmatcha/e2e-skills

Even without the tool, the habit is useful: next time your E2E suite is green, ask which test would actually fail if the feature broke tonight.

Comments

[Aldo]

This hits so close to home. I remember one particular incident where a critical user flow in our SaaS platform, a complex data import process, had a passing E2E test for weeks. The test meticulously clicked through modals, uploaded a file, and waited for a success toast. Everything was green. Then a customer reported their imports weren't showing up, and it turned out the backend service responsible for processing the file had been silently failing for days, but because the UI showed the success toast based on an initial API response, the E2E test passed without issue. We were asserting on the immediate UI feedback, not the eventual system state.

The core challenge isn't just writing tests, but ensuring they're asserting on the right things and truly reflect user-facing success or failure. For us, that meant a conscious shift to not just checking if an element exists, but if it contains the correct, dynamic data that signifies a successful operation, or if a specific error message appears under failure conditions. It's the difference between "did the success message render?" and "did the data actually persist correctly, and can I now see it reflected elsewhere in the application?"

This often pushes us to implement what's almost a manual form of mutation testing: intentionally breaking a dependency or a piece of logic in a staging environment and observing which E2E tests turn red. If none do, that test is giving false confidence. It's a constant trade-off between test speed and the depth of validation, but for critical flows, we lean heavily into assertions that interact with the system's actual state, even if it means a slightly slower or more complex test fixture.