WebDriver vs Chrome DevTools (Speed Test)

Today I got inspired by @FedericoToledo to write a test script that can automate a simple clicking game, but with a focus on executing as fast as possible.

Test automation challenge: Can you prepare a test script to win the game 1to50 automatically as fast as possible https://t.co/H6mwYxeKko???

My solution with Ruby and Selenium took 4.9secs to win the game and I know it can be faster https://t.co/HadbaCzien pic.twitter.com/UrPDh2bfGX

— Federico Toledo (@fltoledo) April 11, 2020

This challenge inspired me to use my favorite Web Testing Framework webdriverIO which support devtools and webdriver protocols out of the box with minor setup. So I decided to try and see which protocol executes faster.

Let's explain the details first.

In order to make your tests scripts execute actions against a website your test framework has to use an interface which is actually capable of controlling the browser.

There are 2 very popular interfaces that allow this to happen. Selenium uses WebDriver (w3c protocol) which is today the most popular one, but there is also Chrome DevTools which is native to chromium browsers and has some benefits and disadvantages over its competitor. In this post we are focusing on speed, and spoiler alert our winner is devTools.

Here is why!

WebDriver sits in the middle between the test scripts and the browser, and uses http traffic to communicate back and forth which can cause a small latency between the execution and the response from the browser.

On the other hand Chrome DevTools allows direct control over the browser by using the Puppeteer library which is finally used by WebDriverio. This means direct execution and less latency on the results.

Let's do the test

The idea of the game is to click the numbers from 1 to 50 as fast as possible, but only the next 25 are displayed. So if you click 1 then the 26 appear. Game

I created a simple script using WebdriverIO on standalone mode so that it runs fast without all the test framework setup.

const { remote } = require('webdriverio');
(async () => {
    const browser = await remote({
        logLevel: 'trace', capabilities: { browserName: 'chrome'}
    })
    await browser.url('https://zzzscore.com/1to50/en/?ts=1592666149743') // navigate
    for(i = 1; i<51; i++){
        var element = await browser.$('//div[text()="@id"]'.replace('@id',i))
        await element.click(); //click each number
    }
    var result = await browser.$('.level');
    console.log(`The result is ${await result.getText()}`);
    // await browser.deleteSession()
})().catch((e) => console.error(e))

By default webdriverIO v6 will try to connect using devtools protocol unless a explicit WebDriver path is specified.

To run the script just type

node <scriptname>.js

DevTools Result

The first thing you will notice when executing the script is this line of log: INFO webdriverio: Initiate new session using the devtools protocol which indicates the used protocol.

And you get this result: The result is 0.85. Which is really fast, it meant at least 50 browser interactions in less than a second.

WebDriver Result

In order to execute the same script on WebDriver protocol, we need to specify the hostname and path when creating the remote object. We also require a running webdriver server. I recommend installing the webdriver-manager package which simplifies the process a lot.

Once the webdriver is running (by default it runs on localhost:444/wd/hub), then just update the code.

...
 const browser = await remote({
        logLevel: 'trace', capabilities: { browserName: 'chrome'},
        hostname: 'localhost', path:'/wd/hub'
    })
...

Again you will notice that the webdriver protocol is specified: INFO webdriverio: Initiate new session using the webdriver protocol.

And the result is: The result is 3.801 which is impressing but it is 4 times slower than the first execution and places webdriver in the second place.

Final Thoughts

We can easily conclude that devtools protocol execute scripts way faster which makes me consider it as a scripting and scraping tool. But there is no free lunch, webdriver is now days the most used interface and the one that has support for cross browser testing, devices and even desktop apps.

I will also mention the great capability of webdriverIO to support both protocols out of the box. It really matters a lot when a tool allows you the power of choice.

Credits for the protocols images to webdriverio website

Comments

[Kevin Lamping]

This is a neat little exercise! I keep trying to think of ways to make it faster, but haven't had any success. I was able to switch to using a CSS-based selector targeting off the 'z-index', but it didn't seem to improve speeds at all. I've heard that xpath is slower than CSS, so I was hopefuly, but seems like it didn't matter that much in this circumstance.

[Pablo Calvo]

Hi @kevin , thanks for commenting out. Yeah I have also heard that xpath strategy is slower than css. But so far it is like a myth, everybody believes it but nobody has seen it.

I think it actually depends on the complexity of the website DOM, but mostly on the xpath strategy: complicated locators with text validations sounds like they might decrease performance. (Not sure until I do an experiment).

Also, I was able to successfully beat the score on this post by injecting the javascript which locates (using javascript xpath locate function) and dispatches the click event using webdriverIO. At some point the execution was faster than the actual website response, so I don't think there can be a faster response.

Check out the code. gist.github.com/pjcalvo/6875f07ec6...

And the score dev-to-uploads.s3.amazonaws.com/i/...

[Kevin Lamping]

You read my mind on using browser.execute to trigger the tap event. I was trying that last night, but couldn't get it to trigger the event on the right element. Nice work figuring it out!

And yeah, I don't think that score is going to be beaten, aside from completely cheating and just bypassing the clicking all together and somehow triggering the "completion" function.

[Pablo Calvo]

I just read this. Awesome reply will dig deeper