E2E tests can pass while testing nothing

Yongjae Lee — Fri, 03 Jul 2026 00:01:12 +0000

Code-in-the-browser, code-server, had an E2E suite that looked healthy. CI was green. The PR had been reviewed. But one test file still contained this:

it.only(...)

That one line quietly disabled the rest of the tests in the file. Eight tests did not run for seven months, while the check still passed. The fix eventually merged in coder/code-server#7845.

This is not a story about code-server being careless. It is almost the opposite: even mature projects, reviewed PRs, and AI-reviewed changes can end up with E2E tests that are green but do not prove the thing they appear to prove.

I mean "testing nothing" in a narrow sense. The test may execute, and CI may pass, but the assertion does not observe the user-visible condition it claims to protect.

I started collecting these cases while fixing Playwright and Cypress tests across open-source repositories. The same shapes kept appearing.

The same shape in other projects

Carbon had an assertion like this (carbon#22564):

expect(page.locator('.cds--progress-step--complete')).toBeTruthy();

page.locator(...) is not a DOM lookup result. It is a Locator object. The object is truthy whether the element exists or not, so the assertion can pass even when the UI is wrong. The intended check is closer to this:

- expect(page.locator('.cds--progress-step--complete')).toBeTruthy();
+ await expect(page.locator('.cds--progress-step--complete')).toBeVisible();

Ghost had a Playwright web-first assertion without await (Ghost#28712). The line looked like an assertion, but it did not wait for the condition it was supposed to verify.

Strapi had this pattern (strapi#26630):

expect(likeButton.isDisabled()).toBeTruthy();

In Playwright, methods such as isDisabled(), isVisible(), and isHidden() return Promise<boolean>. The Promise object itself is truthy, so the assertion can pass regardless of the actual button state. SvelteKit had the same family of issue in kit#16068.

These were not synthetic examples: the broader pass has produced 12 merged upstream fixes so far.

The details differ, but the underlying failure is the same: "there is a test" and "the test can fail for the intended regression" are different claims.

Checks that catch the low-hanging fruit

These are not a complete review. They are just fast checks that have found real problems in real Playwright/Cypress suites.

# 1. Focused tests accidentally committed
rg "(it|test|describe)\.only\(" e2e tests cypress playwright

# 2. Truthiness checks on Locator-like objects or handles
rg "expect\([^)]*\)\.(toBeTruthy|toBeDefined|not\.toBeNull)\(" e2e tests cypress playwright

# 3. Playwright web-first assertions without await
rg "^[[:space:]]*expect\(.+\)\.to(BeVisible|BeHidden|BeEnabled|HaveText|HaveURL|HaveTitle|HaveCount)\(" e2e tests playwright

# 4. State reads whose result is discarded
rg "^[[:space:]]*(await )?[A-Za-z_.]+\.(isVisible|isEnabled|isDisabled|isHidden)\(\)[[:space:]]*;?[[:space:]]*$" e2e tests cypress playwright

The first two are usually low-noise. The third and fourth need context before you automate fixes. But as a first pass, they are useful because they look for tests that can stay green while asserting little.

Some of this belongs in lint rules. I split out the lintable subset into eslint-plugin-playwright-silent-pass and eslint-plugin-cypress-silent-pass. Grep is good for a quick audit; ESLint is better for preventing repeats.

npm i -D eslint-plugin-playwright-silent-pass   # or eslint-plugin-cypress-silent-pass

What general reviewers tend to miss

At first I assumed general AI PR reviewers would catch most of this. To check that, I re-reviewed 100 already-AI-reviewed open-source E2E PRs across 77 repositories. A human-confirmed benchmark found 110 material E2E-test issues in those PRs (results).

The comparison was:

e2e-reviewer: 78 / 110 material E2E issues found, 0 false positives
General AI PR reviewers: 45 / 110 found, 10 false positives

I do not think this means general reviewers are bad. They are trying to review architecture, types, security, performance, style, and product logic at the same time. This problem is narrower. You have to keep asking: can this assertion fail? Does this locator observe the DOM state, or just construct an object? Did an async assertion get dropped on the floor?

About half the cases are not obvious from a grep alone. For example:

expect([200, 401, 403]).toContain(status)

That might be an intentional resilience check, or it might be hiding a real auth regression. You need test context to decide.

What I built

e2e-skills is a small set of agent skills for Playwright and Cypress E2E work. The goal is not to ask "does a test exist?" but "can this test actually fail when the product regresses?"

e2e-reviewer statically reads Playwright/Cypress specs and Page Objects, then checks 24 weak-diagnostic anti-patterns: focused tests, locator truthiness, missing await, discarded state reads, loose status assertions, and related patterns that keep CI green while proving little.
playwright-test-generator is for creating new Playwright coverage. It first looks for coverage gaps, then explores the live app in a browser, drafts tests for real user flows, pauses at an approval gate, and finally reviews the result with the same weak-assertion rules. It is closer to "find the flow worth proving and write fail-capable assertions" than "dump a test file into the repo."
playwright-debugger reads Playwright reports, traces, screenshots, and CI output to classify failures: timing, selector drift, auth/session setup, environment trouble, or a real product regression.
cypress-debugger does the same kind of failure classification for Cypress mochawesome/JUnit output.

The scope is deliberately narrow: Playwright and Cypress only. I would rather catch common fail-open patterns in those two frameworks well than claim shallow support for every browser automation stack.

Repo: github.com/voidmatcha/e2e-skills

Even without the tool, the habit is useful: next time your E2E suite is green, ask which test would actually fail if the feature broke tonight.

DEV Community: Yongjae Lee

E2E tests can pass while testing nothing

The same shape in other projects

Checks that catch the low-hanging fruit

What general reviewers tend to miss

What I built