Capturing browser console logs on automated test

Test flakiness is not only annoying, but also costing time, energy, and money in the long run. That's why it is important to deal with them as soon as possible.

One of the most challenging test type to debug and fix when flakiness occurs is feature test with browser testing. This is because there are usually many elements involved in the test (backend/server, the test spec, and the browser controller) and these elements have to work together in the correct manner for the test to run. Most of the time, flakiness occurs due to race condition (the browser controller was too fast for the backend, etc)

Recently our application, which is built on Ruby on Rails on the backend suffers from test flakiness during CI. Our feature tests are setup to use Capybara with Selenium Chrome driver, and when the test failed, it failed at random points with error message that's too generic. Making things complicated is the fact that tests that succeed locally failed on Travis CI.

Unable to simulate the failing tests, I had to resort to different method: figure out what the browser console output at the time of failure. With only headless mode available on Travis CI, this can be done programatically:

  def console_logs
    driver.browser.logs.get(:browser)
  end

Please note that the selenium-webdrivers gem need to be the newer version (v4 onwards) for this to work.

By default this will only capture log messages on warn/fatal level (output from console.warn and console.error). To retrieve the log messages on info/debug level, I needed to change Selenium Chrome's logging preference option:

Capybara.register_driver :chrome do |app|
  chrome_options = Selenium::WebDriver::Chrome::Options.new

  chrome_options.logging_prefs = {
    browser: 'ALL', # Capture JavaScript errors
    driver: 'SEVERE', # Capture WebDriver errors
  }

  http_client = Selenium::WebDriver::Remote::Http::Default.new

  options = {
    browser: :chrome,
    options: chrome_options,
    http_client: http_client,
  }
  Capybara::Selenium::Driver.new(app, **options)
}

That's it folks. Hope that helps somebody too.