TL;DR: TestFly 1.0.6 and 1.1.0 close three cross-browser reliability gaps: browser.bidi opens a real WebDriver BiDi connection to a locally launched Chrome, Edge or Firefox so ConsoleErrorCollector reads console.error calls and uncaught exceptions from BiDi log events instead of a JS shim — catching errors Firefox used to hide — and ConsoleErrorCollector.isBiDiActive() tells you which capture path ran. RemoteDriverProvider now resolves Edge and Safari on Selenium Grid and honors browser.matrix, where it previously only knew chrome/firefox and always read browser.name. And JVM shutdown hooks plus DriverManager.quitAllTrackedDrivers() guarantee every tracked browser — on every thread, not just the hook thread's own — is quit once when a run is aborted with Ctrl+C or System.exit.
How Does BiDi Catch Console Errors Firefox Used to Miss?
TestFly's captureConsoleErrors has always read browser console errors, but on Firefox — which exposes no browser logs over classic WebDriver — that meant injecting a JavaScript shim that only sees errors after it runs on the current page. An error thrown while the page is still loading, or on a page reached by following a link, was invisible to the shim.
browser.bidi (default false) fixes this at the protocol level: when a Chrome, Edge or Firefox session is launched locally, TestFly asks for a real WebDriver BiDi connection and subscribes to BiDi log events from the moment the test gets its browser:
# testfly.yml
browser:
name: firefox
bidi: true
captureConsoleErrors: true
The capability request is one line, added only when BiDi is configured on:
// BiDiCapability.java
static void applyIfConfigured(MutableCapabilities options, TestFlyConfig config) {
if (config != null && config.getBrowser() != null && config.getBrowser().isBidi()) {
options.setCapability("webSocketUrl", true);
}
}
ConsoleErrorCollector then opens a LogInspector over that connection per session, keyed in a WeakHashMap<WebDriver, BiDiCapture> so sessions are never pinned in memory after the driver is gone:
private static final Map<WebDriver, BiDiCapture> BIDI =
Collections.synchronizedMap(new WeakHashMap<>());
Because the subscription starts at beginTest(driver), errors from a previous test are forgotten even on a reused, per-suite browser — no JS shim, no page-load race.
Safari and remote drivers (Selenium Grid, BrowserStack, Sauce Labs) have no BiDi connection in TestFly today. In that case one warning is logged per session and the framework falls back to the usual capture path — nothing throws.
To know which path actually ran, without reading logs:
if (ConsoleErrorCollector.isBiDiActive()) {
// BiDi log events are driving capture for this session
}
// ConsoleErrorCollector.java
@TestFlyApi(since = "1.1.0")
public static boolean isBiDiActive() {
try {
return BIDI.containsKey(DriverManager.getDriver());
} catch (RuntimeException e) {
return false;
}
}
How Do You Know Which Browser Features You're Trading Away?
A locally started Chrome or Edge gives TestFly the Chrome DevTools Protocol (CDP), which powers network mocking, device emulation, geolocation overrides, and the screencast-based video recorder. Firefox, Safari, Selenium Grid and cloud sessions don't expose CDP — even a remote Chrome doesn't — so those features degrade silently unless something says so.
BrowserFeatures (new in 1.1.0) names the four CDP-dependent features and what happens instead:
public enum Feature {
NETWORK_MOCK("Network interception requires a local Chrome/Edge session... "
+ "routes are ignored and requests reach the real network."),
CDP_RECORDING("Video recording uses periodic screenshots instead of the "
+ "DevTools screencast, so frames are less frequent."),
DEVICE_EMULATION("Device emulation only resizes the window and overrides "
+ "navigator.userAgent in the current page..."),
GEOLOCATION("The location is overridden through JavaScript in the current "
+ "page only and is lost on navigation.");
}
BrowserFeatures.supports(driver, feature) answers the question directly (driver instanceof ChromiumDriver for three of the four, HasDevTools for recording), and warnOnce(driver, feature) logs the fallback exactly once per browser session per feature, naming the browser:
BrowserFeatures.warnOnce(driver, BrowserFeatures.Feature.GEOLOCATION);
GeoLocation already calls this before falling back to its JS override on Firefox. The video recording fallback used to be silent; now every degraded feature says so.
How Do You Run Edge and Safari on Selenium Grid?
Before this release, RemoteDriverProvider only built ChromeOptions or FirefoxOptions and always read browser.name directly — so a browser.matrix run on a Grid used whatever browser.name happened to say, ignoring the matrix browser entirely, and Edge/Safari on Grid wasn't supported at all.
Browser resolution now goes through one shared method that both the local and remote paths call, BrowserContext (the matrix override) winning over YAML:
// DriverProviderFactory.java
public static String resolveBrowserName(TestFlyConfig config) {
String contextBrowser = BrowserContext.get();
if (contextBrowser != null && !contextBrowser.isBlank()) {
return contextBrowser;
}
String name = config.getBrowser().getName();
if (name == null || name.isBlank()) {
throw new IllegalStateException(
"No browser selected: set browser.name, or run through browser.matrix.");
}
return name;
}
RemoteDriverProvider.buildOptions(config, browser) is a public, independently testable method that now switches over all four Grid-supported browsers:
// RemoteDriverProvider.java
switch (browser.toLowerCase(Locale.ROOT)) {
case "chrome" -> options = chromium(new ChromeOptions(), browserConfig.isHeadless(), arguments);
case "edge" -> options = chromium(new EdgeOptions(), browserConfig.isHeadless(), arguments);
case "firefox" -> { /* FirefoxOptions, -headless */ }
case "safari" -> {
if (browserConfig.isHeadless()) {
System.err.println("[TestFly] Safari does not support headless mode via Selenium; ignoring headless=true.");
}
options = new SafariOptions();
}
default -> throw new IllegalStateException(
"Unsupported browser for remote execution: " + browser + ". Supported: chrome, edge, firefox, safari.");
}
options.setCapability("unhandledPromptBehavior", "ignore");
A matrix config that runs the same suite across Chrome, Edge and Safari on a Grid now works unmodified:
# testfly.yml
execution:
mode: remote
gridUrl: http://grid.local:4444/wd/hub
browser:
matrix: [chrome, edge, safari]
A companion fix (do not reuse a per-suite driver for another matrix browser) stops DriverManager from handing a Chrome per-suite driver to a thread that's supposed to be testing Edge — DRIVER_BROWSER_CONTEXT is now tracked per thread and checked before reuse. Another (quit the browser when post-launch session setup fails) makes every built-in provider quit the session before rethrowing when timeout configuration throws right after launch, so a setup failure no longer leaks a browser process.
How Does TestFly Guarantee Browsers Are Never Left Running?
DriverManager has registered a JVM shutdown hook since early on, but until this release it only quit per-suite drivers and whatever happened to be bound to the shutdown hook's own thread — which is never the thread that actually ran a per-test driver. Abort a parallel per-test run with Ctrl+C or System.exit, and worker-thread browsers kept running after the JVM exited.
The hook now calls quitAllTrackedDrivers(), which unions every driver that still holds a session permit with every registered per-suite driver, across all threads, and quits each exactly once:
static void quitAllTrackedDrivers() {
java.util.Set<WebDriver> drivers =
java.util.Collections.newSetFromMap(new java.util.IdentityHashMap<>());
drivers.addAll(PERMIT_HOLDERS);
drivers.addAll(SUITE_DRIVERS);
for (WebDriver driver : drivers) {
try {
driver.quit();
} catch (Throwable ignored) {
} finally {
SUITE_DRIVERS.remove(driver);
releasePermitOf(driver);
}
}
}
PERMIT_HOLDERS is the set of drivers currently holding a session-slot permit — populated and cleared by exactly one code path per driver, which is also what makes execution.maxActiveSessions leak-proof. Normal suite teardown still calls DriverManager.quitAllSuiteDrivers() then quitDriver() from SuiteExecutionListener, TestFlyExtension (JUnit 5) and CucumberHooks; the shutdown hook is the safety net for abnormal exits, firing identical cleanup logic no matter which thread owned the browser.
This traces back to 1.0.6's original "Guaranteed Driver Cleanup" line (shutdown hooks plus @AfterAll listener execution): the hook existed from 1.0.6, this release makes it actually reach every thread's browser instead of only the suite-level ones.
FAQ
Does browser.bidi work with Selenium Grid or BrowserStack/Sauce Labs?
No. BiDi only applies to a locally launched Chrome, Edge or Firefox. On Safari or any remote driver, TestFly logs one warning and falls back to the previous capture path (CDP logs on Chrome, the JS shim elsewhere); nothing fails.
How do I check which console-error capture path is active for the current test?
Call ConsoleErrorCollector.isBiDiActive(). It returns true only when the current thread's driver has an open BiDi log subscription; it never throws, returning false if there's no driver or the check fails.
Can I run the same suite on Chrome, Edge and Safari against one Selenium Grid?
Yes, with execution.mode: remote, execution.gridUrl and browser.matrix: [chrome, edge, safari]. RemoteDriverProvider resolves the matrix browser ahead of browser.name and builds Grid-appropriate options for all four supported browsers, including Safari's headless restriction.
If I kill a parallel run with Ctrl+C, are all browser processes definitely gone?
Yes. The JVM shutdown hook now collects every driver that holds a session permit plus every per-suite driver, on every thread, and quits each one exactly once, regardless of which thread the hook itself runs on.
Top comments (0)