DEV Community

Vladimir Elchinov for Session Replay

Posted on

Your End-to-End Tests Run a Chrome With Popup Blocking and Background Throttling Turned Off

A user reports a bug. You write a Playwright or Puppeteer test that does exactly what they did. It passes. You close the report.

Before you do, look at how your test runner starts Chrome, because it is not the way your user starts it. Both tools launch the browser with a list of default switches, and several of those switches turn off the precise behaviours that produce bugs users report and developers cannot reproduce.

What the defaults are

From Puppeteer's ChromeLauncher.ts and Playwright's chromiumSwitches.ts, current main of each:

Both of them pass --disable-popup-blocking, --disable-prompt-on-repost, --disable-background-timer-throttling, --disable-backgrounding-occluded-windows and --disable-renderer-backgrounding.

Playwright also passes --disable-extensions and --disable-back-forward-cache.

Puppeteer also passes --enable-automation.

Every one of those is a reasonable choice for a test runner. A suite needs to be deterministic, and a browser that blocks a popup sometimes, throttles a timer sometimes and restores a page from cache sometimes is not. The trouble is only that your users get the "sometimes".

The bug each one hides

--disable-popup-blocking. Chrome lets a page open a window in response to a click. A window.open that runs after an await, once the click has been used up, is the classic one that works in development and is blocked in the field. With popup blocking off, it is never blocked in your tests either.

--disable-prompt-on-repost. The "Confirm Form Resubmission" dialog, which a user meets when they reload or go back to a page that was the result of a POST, does not appear. Whatever your page does after it is untested.

The three background switches. Chromium's own description of --disable-background-timer-throttling is "Disable task throttling of timer tasks from background pages", and of --disable-renderer-backgrounding that renderers "are never told that they are backgrounded or hidden" at the operating system level. A user with your app in a background tab gets a setInterval that fires far less often than you asked. Your test gets the interval you wrote.

--disable-back-forward-cache (Playwright). The back-forward cache restores a page exactly as the user left it when they press Back, without running your load code again. That is where "I pressed Back and the basket showed the old total" comes from. Playwright's source gives its reason in a comment: it "Avoids surprises like main request not being intercepted during page.goBack()". Sensible for a test, and it means no Playwright test of yours has ever met a page restored from that cache.

--disable-extensions (Playwright). No ad blocker, no privacy extension, no password manager. The requests those block, and the DOM they change, are not in your test's world, and they are in most of your users'.

--enable-automation (Puppeteer). This one switches off something specific. Chrome normally reloads a network error page by itself, after one second, then five, then thirty, and the code that installs that behaviour checks for this switch and leaves it out when it is present. A user sees an error flash and go away. Your test sees it stay.

And under all of these, a fresh profile: no saved passwords, no cookies from last month, no history. That on its own is the difference between a login form a returning user sees already filled in and the empty one your test types into.

What to do about it

Keep the defaults for the bulk of the suite. The determinism is worth it.

Then add a small set of tests that run a Chrome closer to your users', for the specific behaviours your app depends on. Both tools take an ignoreDefaultArgs option that removes named switches:

// Playwright
const browser = await chromium.launch({
  ignoreDefaultArgs: ['--disable-back-forward-cache', '--disable-popup-blocking'],
})

// Puppeteer
const browser = await puppeteer.launch({
  ignoreDefaultArgs: ['--disable-popup-blocking', '--disable-background-timer-throttling'],
})
Enter fullscreen mode Exit fullscreen mode

A test that opens a popup after an await, one that presses Back on a page with state, one that leaves a timer running in a hidden tab: three tests, each aimed at a class of report you would otherwise close as "cannot reproduce".

The general version

"It passes in my test" is a statement about the browser your tool built, not the one your user has. When a report will not reproduce, the first thing worth comparing is not their steps against yours. It is their browser's configuration against your test runner's launch arguments, and the runner's list is printed in its own source.

Top comments (0)