DEV Community: Pablo Calvo

3 Successful Test Runs Is the Magic Number

Pablo Calvo — Tue, 30 Jul 2024 20:58:16 +0000

(But Is It Really?)

As developers and testers, we've all been there: the excitement of seeing that green checkmark after a successful test run. It feels like a job well done, and the temptation to move on is strong. However, I’m here to tell you that one successful test run doesn't mean your work is finished.

The Pitfall of One Successful Test Run
Relying on a single successful test run to validate your code is a risky practice. A myriad of issues can arise that only become apparent with repeated testing:

Unstable Environments: Minor fluctuations in your testing environment can cause a test to pass once but fail subsequently. This could be due to network instability, server loads, or other environmental factors.
Data Changes: The test data might change, causing subsequent runs to fail. For instance, if your test relies on a particular data set, any alterations can lead to inconsistencies.
Configuration Variations: Sometimes, different configurations or slight changes in the deployment settings can affect the test outcome. Ensuring your test runs successfully across multiple configurations is crucial.

Why Three Test Runs?
So, why do some advocate for three successful test runs? The idea is to introduce a buffer against the aforementioned issues. If your code and tests are robust enough to pass three consecutive times, it’s less likely that flukes or transient issues are at play.

However, while three successful test runs can provide a higher degree of confidence, they should not be seen as a definitive measure of success. It's essential to recognize that the number three is somewhat arbitrary and not inherently magical. The key takeaway should be the principle behind it: consistency.

The True Measure of Success: Consistent Stability
To ensure your code is genuinely ready, focus on the consistent and repeatable success of your tests. Here are some best practices to follow:

Automated Testing Pipelines: Integrate your tests into automated pipelines that run tests multiple times across different environments and configurations. This helps catch issues that might only appear under specific conditions.
Continuous Integration/Continuous Deployment (CI/CD): Implement CI/CD practices to ensure that tests are run frequently, ideally with every code change. This continuous feedback loop helps identify issues early.
Comprehensive Test Coverage: Ensure your tests cover a wide range of scenarios, including edge cases and potential failure points. This reduces the likelihood of unforeseen issues cropping up in production.
Monitoring and Alerts: Post-deployment monitoring can help catch issues that might not have been apparent during testing. Set up alerts for any anomalies or errors in the live environment.

Conclusion

In summary, don’t fall into the trap of believing in a magic number. Instead, aim for the consistent reliability of your tests to ensure your code is truly ready for prime time. Happy testing!

UI Test Automation Framework proposal with SeleniumBase!

Pablo Calvo — Tue, 08 Sep 2020 17:30:12 +0000

Considering to use Seleniumbase.io for my next web automation framework I decided to come up with an initial architecture diagram that illustrates the integration of the framework with the rest of components. Using Python Webdriver (SeleniumBase) and pyTest.

In this case we will be using the PageObjectModel which is commonly used as a first approach for web testing frameworks:

So let's elaborate a little bit further on each of the specific components:

SeleniumBase

Will serve as our default WebDriver wrapper, which will be in charge of most of the UI interactions, assertions and eventually reporting and logging.

Utilities/WebDriverWrapper

Will be used to wrap those custom UI interactions that are not supported by SeleniumBase, but still will be used across many of the pages and test cases.

DataWrapper

Will centralize the test generation and serving process for all the test cases and keywords (common flows), so that we ensure that no hardcoded data will be generated at different stages of the test process.
The data could be anywhere from api services, json files or random generated data.

Keywords (common flows)

Will contain user flows or actions that need interaction with at least 2 pages objects and that is commonly used across different tests. This will help us reduce code duplicity and eventually build a repository of keywords that could function as blocks to easy build new test cases.

Base Page and Page Objects

BasePage will initially just be added as a placeholder in case we require it. Notice that not all the pages inherit from it, as we want to avoid the jungle inheritance problem.

The page objects will follow the pattern of providing locators and user actions that are available for specific pages.
Note: We could eventually split the locators under their our classes, but initially that would be an overhead.

Base Test Case and Test Cases

Finally our test cases layer. At this point is where we are going to define our test cases structure based on actions or elements delivered by each of the PageObjects. Also at this layer only is where test assertions are executed.

Final thoughts

Of course there is a lot of room for changes with this approach and we will be seeking for feedback on improvements. But overall based on lessons learned and trying to follow a single responsibility principle we think this initial architecture will let us:

Start writing valuable test cases soon.
Avoid over engineering issues.
Heavily rely on SeleniumBase to drive most of the element interactions.
Be mindful about classes responsibilities and code re-usabilities problems.

Anything else we are missing.

Problems with webDriver JavaScript clicking!

Pablo Calvo — Wed, 02 Sep 2020 18:11:41 +0000

"Let's click that element using javascript... so we could avoid issues like ElementNotInteractable, or AnotherElementWillReceiveTheClick." some automation engineer - 2020.

I have heard this one many times, and to be honest there are edge cases where it is the only way to make that click work. But I personally try to use that approach as the latest source and here is why...

Story

I got a message from a customer today saying: 'Hey marketing can't save new components on the page, they are saying that there is no save button'.

So I opened my email to check the automated tests and they were all Passed. 100% success. At this point I started to think that it was most likely a glitch on the system. So I started testing manually and found out that the save icon was indeed missing.

But then.. how did my tests didn't find this issue? This is a flow that all the tests go through, that means I should have had hundreds of failures.

So I opened the code and.. found the problem right away...



...
seleniumHelper.clickElementJavaScriptExecutor(checkIcon);
...
   public void clickElementJavaScriptExecutor(WebElement element){
        ((JavascriptExecutor)driver).executeScript("arguments[0].click();", element);
    }
...

This is one of those cases where the bug is 100% test case proof, or in different words I had useless tests.

Explanation

For this specific case, the element does exist on the page.

But it was hidden, not visible for normal customers and for javascript that doesn't mean anything. The script will just trigger the click event bind to it and move on as long as the element exist.

And that caused the automated flows to complete with no error, the test cases never exposed a real customer issue. After changing the javascript clicking for a normal click, all the tests started failing as expected.



checkIcon.click();

WebDriver error. Failed Authoring: Component: phonenumber
Failure Reason: org.openqa.selenium.ElementNotInteractableException: element not interactable.

End notes

Remember that every single action on the script should behave as a validation. Clicking, typing, selecting.. if one of those fail you got yourself a problem in the website this is a continues validation approach that would help your tests being more reliable and avoid false positives. (like in my story).

So using javascript to click the elements is sometimes a good idea, for example when the normal webdriver clicking doesn't seem to work, or you have a very specific UI that place components on top of each other and the clicking area is very small.

But we should try to avoid it, cause it is not a real user flow.

Comments?

cheers :) and remember to keep using webdriver
https://pjcalvo.github.com

if you liked the post:

WebDriver vs Chrome DevTools (Speed Test)

Pablo Calvo — Sat, 20 Jun 2020 20:18:44 +0000

Today I got inspired by @FedericoToledo to write a test script that can automate a simple clicking game, but with a focus on executing as fast as possible.

Test automation challenge: Can you prepare a test script to win the game 1to50 automatically as fast as possible https://t.co/H6mwYxeKko???

My solution with Ruby and Selenium took 4.9secs to win the game and I know it can be faster https://t.co/HadbaCzien pic.twitter.com/UrPDh2bfGX
— Federico Toledo (@fltoledo) April 11, 2020

This challenge inspired me to use my favorite Web Testing Framework webdriverIO which support devtools and webdriver protocols out of the box with minor setup. So I decided to try and see which protocol executes faster.

Let's explain the details first.

In order to make your tests scripts execute actions against a website your test framework has to use an interface which is actually capable of controlling the browser.

There are 2 very popular interfaces that allow this to happen. Selenium uses WebDriver (w3c protocol) which is today the most popular one, but there is also Chrome DevTools which is native to chromium browsers and has some benefits and disadvantages over its competitor. In this post we are focusing on speed, and spoiler alert our winner is devTools.

Here is why!

WebDriver sits in the middle between the test scripts and the browser, and uses http traffic to communicate back and forth which can cause a small latency between the execution and the response from the browser.

On the other hand Chrome DevTools allows direct control over the browser by using the Puppeteer library which is finally used by WebDriverio. This means direct execution and less latency on the results.

Let's do the test

The idea of the game is to click the numbers from 1 to 50 as fast as possible, but only the next 25 are displayed. So if you click 1 then the 26 appear. Game

I created a simple script using WebdriverIO on standalone mode so that it runs fast without all the test framework setup.

const { remote } = require('webdriverio');
(async () => {
    const browser = await remote({
        logLevel: 'trace', capabilities: { browserName: 'chrome'}
    })
    await browser.url('https://zzzscore.com/1to50/en/?ts=1592666149743') // navigate
    for(i = 1; i<51; i++){
        var element = await browser.$('//div[text()="@id"]'.replace('@id',i))
        await element.click(); //click each number
    }
    var result = await browser.$('.level');
    console.log(`The result is ${await result.getText()}`);
    // await browser.deleteSession()
})().catch((e) => console.error(e))

By default webdriverIO v6 will try to connect using devtools protocol unless a explicit WebDriver path is specified.

To run the script just type

node <scriptname>.js

DevTools Result

The first thing you will notice when executing the script is this line of log: INFO webdriverio: Initiate new session using the devtools protocol which indicates the used protocol.

And you get this result: The result is 0.85. Which is really fast, it meant at least 50 browser interactions in less than a second.

WebDriver Result

In order to execute the same script on WebDriver protocol, we need to specify the hostname and path when creating the remote object. We also require a running webdriver server. I recommend installing the webdriver-manager package which simplifies the process a lot.

Once the webdriver is running (by default it runs on localhost:444/wd/hub), then just update the code.

...
 const browser = await remote({
        logLevel: 'trace', capabilities: { browserName: 'chrome'},
        hostname: 'localhost', path:'/wd/hub'
    })
...

Again you will notice that the webdriver protocol is specified: INFO webdriverio: Initiate new session using the webdriver protocol.

And the result is: The result is 3.801 which is impressing but it is 4 times slower than the first execution and places webdriver in the second place.

Final Thoughts

We can easily conclude that devtools protocol execute scripts way faster which makes me consider it as a scripting and scraping tool. But there is no free lunch, webdriver is now days the most used interface and the one that has support for cross browser testing, devices and even desktop apps.

I will also mention the great capability of webdriverIO to support both protocols out of the box. It really matters a lot when a tool allows you the power of choice.

Credits for the protocols images to webdriverio website

Emulate GeoLocation for Automated Testing with webdriverIO

Pablo Calvo — Fri, 01 May 2020 02:57:02 +0000

As exposed on the first post of this series, Google Chrome DevTools includes a series of features that facilitate the emulation of customers Experience Profiles, such as device, resolution and of course geographical location.

Chrome DevTools Protocol

Once you have mastered the art of manual testing and want to try some automated geoLocation testing, you don't have to look away looking for new tools. Chrome DevTools exposes its own Protocol which allow us to programmatically use the same tools you know how to use on the browser.

Let's automate it

For the sake of this post I decided to use webdriverIO to write the automated tests. The reason to use this framework is that there is 3rd party service already developed and ready to use that allow us to seamlessly use the devTools without adding any extra code.

Getting started

Setup the project

We need to create a folder, and start a nodeJs project. (Recommended: Make sure the latest version of node is installed).



$ mkdir webdriverio-test && cd webdriverio-test
$ npm init -y

Install webdriverIO CLI

Now let's install the wdio CLI, which provide us with tools to setup and run a basic webdriverIO project.



$ npm i --save-dev @wdio/cli

Generate a wdio config file and create a test folder

WebdriverIO uses a configuration file (by default wdio.conf.js) that contains the basic information of the framework. Once the basic setup is complete webdriverIO will look into ./test/specs for the test files.



$ npx wdio config -y
$ mkdir -p ./test/specs

Setup devTools service

So far we have only setup the basic default configuration for webdriverIO. On this next step we are going to include the devTools service which will provide us with capabilities to interact with the devTools protocol.



$ npm install @wdio/devtools-service --save-dev

Then go to the wdio.conf.js and add the devtools service to the webdriverIO configuration.



...
// Services take over a specific job you don't want to take care of. They enhance
// your test setup with almost no effort. Unlike plugins, they don't add new
// commands. Instead, they hook themselves up into the test process.
    services: ["devtools"],
..

This is basically all the necessary configuration

Now let's create some tests

Create a file under the folder above and name it geo.js. This file will contain our testing code.

Test Scenario

Navigate to https://www.where-am-i.net/ and verify the current position.
Use devTools to emulate a different location. (London)
Verify that the user is on a new location.
Again use devTools to emulate another location. (Tokyo)
Repeat the verification.

Read the comments in the code



const assert = require('assert')

describe('Check Location overriding the timezone', () => {
    it('Check the default timezone', () => {
        browser.url('https://www.where-am-i.net/')
        // this validation depends on where are you!
        // assert.equal(location.getText(), '19.075984, 72.877656')
        // you might get 0.0 and 0.0 the first time because of authorization issues
    })

    it('Check that London should be the timezone', () => {
        // change to london location and timezone
        // cdp is the command to use a devTools command
        // we are going to use the Emulation module
        // and the setGeoLocationOverride method
        // info: https://chromedevtools.github.io/devtools-protocol/tot/Emulation/
        // note: the location and timezone must match or you will get an unavailable position    

        browser.cdp('Emulation', 'setGeolocationOverride' , {
            latitude:51.507351,
            longitude:-0.127758,
            accuracy: 1
        })
        browser.cdp('Emulation', 'setTimezoneOverride', {
            timezoneId: 'Europe/London'
        })       
        browser.pause(3000) // wait so you can notice the map changing
        let location = $('#location') // get a location reference for validation
        assert.equal(location.getText(), '51.507351, -0.127758')

    })

    it('Check that Tokyo should be the timezone', () => {
        // change to lo Tokyo location and timezone
        browser.cdp('Emulation', 'setGeolocationOverride' , {
            latitude:35.689487,
            longitude:139.691706,
            accuracy: 1
        })

        browser.cdp('Emulation', 'setTimezoneOverride', {
            timezoneId: 'Asia/Tokyo'
        })
        browser.pause(3000) // wait so you can notice the map changing
        assert.equal($('#location').getText(), '35.689487, 139.691706')
    })
});

Let's execute the tests

Just run on your terminal



$ npx wdio wdio.conf.js

Just like that!! We just emulated different user locations with no need to add complicated code or any extra tooling.

Now what?

Well... this example is just a very small fragment of what you actually accomplish with devTools and webdriverIO. I encourage you to explore more the documentation for both tools and discover new capabilities that can be added to your tests.

webdriverIO devTools Service
Chrome devTools Protocol

Share your thoughts and comments on how you test geolocation within your automated tests!!

Emulate GeoLocation for Manual Testing

Pablo Calvo — Wed, 29 Apr 2020 15:54:46 +0000

In a web world so advanced that it is almost mandatory to deliver unique User Experiences to unique Customers Profiles, we need to make sure our system behaves as expected under different circumstances, and those circumstances can be as simple as where my customers are located.

Old fashion way!

One simple solution to test Geo Location features is to use a VPN provider that can easily transport us to almost anywhere in the world. But as simple as the solution looks like, it also comes with several constraints: Limited to the VPN provider location points and really hard to programmatically change location on demand, and of course the monetary cost of acquiring the service.

Google Chrome DevTools to the rescue!

With google chrome devTools we can easily override the user current geo location and test our unique experience. Check it out here.

Let's try it out!

Once you have followed the previous link and know how to change the default location, let's make some tests around it.

Navigate to https://www.where-am-i.net/, and check the current location.
Let's override the default user location to Berlin, and notice how the map is automatically updated.
Let's try one more time with a different city.

So far pretty awesome...

Custom locations

We can basically add anywhere in the world as a custom location, as long as the timezone matches with the geo coordinates.

Check this list to find the supported timezones.

First find a timezone you want to test. i.e. America/Costa_Rica.
Go to google maps and find the right coordinates. ..www.google.com/maps/place/San+Jos%C3%A9,+Costa+Rica/@**9.9314988,-84.0951252**,14.76z/data..
Then add everything as a new custom location on the Sensors window.

As simple as that you can know use your custom timezone.

Final Thoughts

Geo Location testing just turned very simple with Google Chrome and devTools. Of course you are limited to this specific browser and I think that the use of a VPN will be a solid complement for this kind of testing.

Comment below what you do to test geo location features on your website, and any comments you have!!

Worst fear working on a Test Automation Project

Pablo Calvo — Wed, 15 Apr 2020 22:00:34 +0000

Every coding project has embedded risks and challenges that are constantly transforming and requires proper care. I have worked in test automation projects for more than 10 years now, I have used many languages, worked in multiple domains, being part of big and small teams and there is one thing that still scares me from time to time.

And it is To build something useless ...

You can smell that something is wrong when you have been developing your test framework for over six months and it is not providing any value to the product.

Purposes of test automation

I truly believe that a test automation framework serves 2 main purposes:

Provides trust and confidence that if no bugs are found and no alerts have been triggered your system is continuously working expected even when new features have been added. This means that your framework serves as a confidence tool and if there have been month without a single test case reporting failure then you can always feel safe.
Provides details when something goes wrong so that you and your team can analyze and take decisions based on the reporting failures.

Challenges of test automation

There are many challenges that we can face when working on a test automation project. The tests could be flaky and report false negatives that will consume time on debugging misleading scenarios. You can also skip important validations and report false positives which will lead into production issues and the possibility of customer facing problems. You can also be limited by technology, stack and setup and have an incomplete framework that only tests partial scenarios which takes us to the previous challenge.

But there is one challenge that I fear the most, and is to build a test automation framework that ends up being useless. Imagine yourself working for three month building the framework structure, you spend time adapting some cool reporting tools, and you use the very best SOLID programming principles to feel confidence about your code, but... nobody is benefiting from the tests. And here are some of the reasons why this could happen:

They take too long to execute.
They are flaky.
There is no CI/CD process within your project.
There is no confidence in the tests.
They are complex to execute on different environments.
There are unrealistic expectations about the testing framework.
Nobody know about them.

What to do?

When you find yourself working on a test framework that is not providing a lot of value to anybody what you need to do is to remember that it is a software project. So what? What I mean is that your project should have stakeholders, should have requirements and a defined release lifecycle.

What you need to do is simple. Just raise your hand to the QA Team, the Dev team, the Product Owners, Project Managers and any other stakeholder that you think should be involved with the project. Ask the right questions and push for guidance, maybe the Product Owner just need one or two critical scenarios tested, while developers might not even understand the benefits of what you are working on and don't get me started with the amount of time testers don't know they could save.

Don't be afraid to start over, it is always better to have ten initial stable test cases and add coverage slowly, than pushing 100 flaky tests which means a lot of debugging and all the trust compromised.

Conclusion

It is easy to lost motivation when the work you are doing is not being appreciated, so make your self valuable and be smart about it.

follow me if you need help implementing your test automation.
https://coffeestainio.github.com

Selenium vs The World Faster Clicker

Pablo Calvo — Wed, 29 Jan 2020 16:29:53 +0000

Today I discovered that a guy named Jordan Hum is the world record holder for more clicks in 5 seconds, with an amazing score of 14 CPS which means 70 clicks in 5 seconds.

So I went to Click Speed Test and tried my self.

You’re a Rabbit! Well.. You Clicked with the speed of 9.6 CPS (Clicks per seconds). You made 48 Clicks in 5 Seconds.

Automating the clicks

I wondered how fast can selenium click? I am pretty sure that a selenium script can beat this guy, but let's make sure finding some results.

This is the script:



driver = webdriver.Chrome(DRIVER_LOCATION) #driver
driver.get("https://clickspeedtest.com/5-seconds.html")
print('Start clicking')
id = driver.find_element_by_id('clicker')
while True: # click for ever
  try:
    id.click() 
  except Exception as ex: # until it breaks
    print('Time is over')
    break

time.sleep(1) # results are slow
result = driver.find_element_by_css_selector('.times')
print(f'Result: {result.text}\n') 
driver.close()

I executed it several times to find an average value, but basically this was the result:



Start clicking
Time is over
Result: 190 Clicks in 5 Seconds

Let's learn something

I wanted to have a lesson learned out of this post, so I tried and tested a couple of myths:

1) The original script finds the element once and then uses that instance to execute the clicks. But what if we try to find the element inside the loop and click it right away driver.find_element_by_id('clicker').click().

The outcome is way slower clicking as expected:



Result: 137 Clicks in 5 Seconds

2) We can try to use a javascript executor to performs the clicks instead of the click method:



javaScript = "document.getElementById('clicker').click();"
driver.execute_script(javaScript);

My lesson learned for this case is that using javascript not always trigger the click event.

3) What if I use a headless browser? Does it makes any difference:

The result is (surprisingly for me) an average faster clicking:



Result: 216 Clicks in 5 Seconds

I executed this a lot of times and in some cases it went down to ~170, but average was mostly faster.

Final thoughts

Maybe clicking this fast is not a normal use case for selenium, but knowing how to improve tests speed and performance can save some minutes of execution on large amounts of tests.

Suggestions on what else we can test?

cheers :) and remember to keep clicking
https://pjcalvo.github.com

if you liked the post:

Revamp Selenium: The power of explicit waits

Pablo Calvo — Fri, 17 Jan 2020 21:15:26 +0000

Working and testing with selenium will make us very patient developers. We fill our code with a lot of waits all the time, and is natural and is expected.

UI tests have a huge dependency on the external conditions of the system under tests, for example: rendering speed, internet connection and third party tools. Also sometimes we need to wait because it is how our system behaves and we need certain conditions to be full-filled before our test can continue executing.

Selenium provides 2 ways of waiting (+1 non selenium):

Implicit wait: Just gives the driver a range of time to find an element before breaking the execution. In natural words: Hey selenium if you can't find my elements in 30 seconds please finish the test.
Explicit wait: Verifies for conditions of an element to be full-filled before breaking or continue with the execution. Hey selenium, find my element but also make sure it has as a class before continue.
Sleep: We sleep our execution thread, then proceed. Hey selenium STOP . . . . . . . . please proceed.

Talking about explicit waits

Imagine we are working with a countdown website, and we want to make sure that the counter actually works.

driver = webdriver.Chrome(DRIVER_LOCATION) #driver
driver.get("https://gcalvocr.github.io/countdown-app/")

print('**** Click on 1 Minute ****')
driver.find_element_by_id("btn-1").click() # click on start 1 minute countdown

At this point we could be tempted to add a 60 seconds hardcoded wait and they verify that the counter is 00:00.

time.sleep(60000)
if driver.find_element_by_id('countdown').text == '00:00':
  print('Yes, the counter worked')
else:
  print('Nop, The expected value is not correct')

And the solution works as a charm.
.

But, what if it fails? We would have to wait 1 entire minute to figure it out. What if our element is not visible anymore after 2 seconds? What if the counter never started?

Let's create a custom explicit wait

class element_has_text(object):
  def __init__(self, locator, text): #python contructor
    self.locator = locator
    self.text = text

  def __call__(self, driver):
    element = driver.find_element(*self.locator) # find the element
    if element.text == self.text: # if the element text is expected return true
        return True
    else:
        return False

The code above extends the WebDriverWait class to add a new custom wait. This custom wait is going to verify constantly for the condition if element.text == self.text, but will break the test only when the given time is burned down and the condition was not met.

So the new code could look like this using the WebDriverWait and the custom element_has_text.

  print('**** Click on 1 Minute ****')
  driver.find_element_by_id("btn-1").click() # click on start 1 minute countdown

wait = WebDriverWait(driver, 10) # wait max 10 seconds
locator = (By.ID, 'countdown') # element locator
wait.until(element_has_text(locator, '00:55'))
print('**** Timer has 55 seconds left ****')
wait.until(element_has_text(locator, '00:45'))
print('**** Timer has 45 seconds left ****')
wait.until(element_has_text(locator, '00:35'))
print('**** Timer has 35 seconds left ****')
# this will break now
wait.until(element_has_text(locator, '00:10'))

And the outcome is a sweet selenium.common.exceptions.TimeoutException that we can handle as part of the test failure:

This is quite more powerful, because it is also looking for the element and verifying a property. In case the element can't be found the test will also break.

Final thoughts

Selenium has several built in explicit waits that will adapt to most of the use cases, so there might no be need for you to create a custom one. But I wanted to go a step further and demonstrate that you can easily extend your testing framework and adapt to your system under test.

Note: I don't want to say that sleep.wait() is the devil and that you should try to avoid it, but yes it is the devil and you should avoid it as much as possible.

Please take your time to read the selenium documentation and reach out if you need any advice on how to improve your code quality and tests speed.

Waits

Comments?

cheers :) and remember to keep waiting
https://pjcalvo.github.com

if you liked the post:

Discussion: Challenges that we face as dev test engineers!?

Pablo Calvo — Mon, 13 Jan 2020 20:29:55 +0000

I have worked on projects where the customer literally requests: We need to automate the testing for this project!

Basically, this makes me ask:

What do you exactly need me to do?

What are the goals?
What are the priorities?
How do you need me to integrate with the SLC!?

What other challenges do you face as test engineers?

Playing around with selenium and openCV!

Pablo Calvo — Thu, 09 Jan 2020 18:30:39 +0000

Selenium library is frequently used for UI test automation, and was probably designed for it, but selenium itself is not a testing library and people quite often forget about it

Selenium is basically a set of tools that allow your code to interact with a browser and the html elements on it.

The selenium WebDriver architecture consists in four main parts:

The language bindings (libraries for each language).
The selenium server.
The browser drivers.
Finally, the drivers.

Playing around with selenium

Selenium is not a super fast processing tool, maybe because of the many components that need to get involved. I made a fun experiment to see how slow is really selenium by reducing the external dependencies at its minimum.

For the experiment I will basically be executing clicks against a system in which no internet call is executed on each click and also there is no html rendering happening (or this does not matter that much).

Python and open CV

I used python along with openCV to open an image and generate a list of pixel points for each black pixel on the given image.

def generate_coordenates():
    print('Generating coordinates based on image')
    image=cv2.imread('image2.png')
    black_coordinates = [] #initial position

    for i in range(image.shape[0]):
        for j in range(image.shape[1]):
            # find the black pixels
            if image[i,j,0]==0 and image[i,j,1]==0 and image[i,j,2]==0: 
                black_coordinates.append((j,i))

    print('Writing coordinates to file')
    # generate a coordinates file
    with open(COORDINATES_FILE, 'a') as out:
        for x, y in black_coordinates:
            out.write(f'{x},{y}\n')

    print('Finish generating coordinates file')

Selenium and a website like `MS Paint`

Then I use selenium to go to a website like MSPaint (https://kleki.com/) and generate an image based on the black pixels.

driver = webdriver.Chrome(DRIVER_LOCATION)
        with open(COORDINATES_FILE) as file:
            driver.get(DRAW_WEBSITE)
            driver.find_element_by_css_selector("html")

            # move to starting point
            actions = ActionChains(driver)
            actions.move_by_offset(STARTING_COORDINATES.get('x'), STARTING_COORDINATES.get('y'))
            # start painting
            for line in map(lambda line: line.rstrip('\n'), file):
                print('Creating move actions')
                xy = line.split(',') # split the x and y
                current = {'x': int(xy[0]), 'y': int(xy[1])} 
                move_to = {'x': current.get('x') - before.get('x'), 
                           'y': current.get('y') - before.get('y')}

                actions.move_by_offset(move_to.get('x'), move_to.get('y'))
                actions.click()
                before = current.copy()

            # finish painting
            actions.perform()

The process

A couple of minutes:

20 minutes:

50 minutes:

The results

The coordinates file ended up with 3777 rows. Because for each row selenium executes 2 actions: move_by_offset and click it means 7554 selenium actions were be executed.

The test took 54.67 minute to complete which means around 3280 seconds.

And if we do the math: 7554 / 3280 = ~2.30. It means that 2.30 selenium actions were executed per second.

Quite fun...

Comments?

cheers :) and remember to keep using selenium
https://pjcalvo.github.com

if you liked the post:

How slow is selenium?

Pablo Calvo — Wed, 08 Jan 2020 19:42:28 +0000

DEV Community: Pablo Calvo

3 Successful Test Runs Is the Magic Number

UI Test Automation Framework proposal with SeleniumBase!

SeleniumBase

Utilities/WebDriverWrapper

DataWrapper

Keywords (common flows)

Base Page and Page Objects

Base Test Case and Test Cases

Final thoughts

Problems with webDriver JavaScript clicking!

Story

Explanation

End notes

WebDriver vs Chrome DevTools (Speed Test)

Let's explain the details first.

Here is why!

Let's do the test

DevTools Result

WebDriver Result

Final Thoughts

Emulate GeoLocation for Automated Testing with webdriverIO

Chrome DevTools Protocol

Let's automate it

Getting started

Setup the project

Install webdriverIO CLI

Generate a wdio config file and create a test folder

Setup devTools service

Now let's create some tests

Let's execute the tests

Now what?

Emulate GeoLocation for Manual Testing

Old fashion way!

Google Chrome DevTools to the rescue!

Let's try it out!

Custom locations

Final Thoughts

Worst fear working on a Test Automation Project

Purposes of test automation

Challenges of test automation

What to do?

Conclusion

Selenium vs The World Faster Clicker

Automating the clicks

Let's learn something

Final thoughts

Revamp Selenium: The power of explicit waits

Talking about explicit waits

Let's create a custom explicit wait

Final thoughts

Discussion: Challenges that we face as dev test engineers!?

What do you exactly need me to do?

Playing around with selenium and openCV!

Playing around with selenium

Python and open CV

Selenium and a website like MS Paint

The process

The results

How slow is selenium?

Selenium and a website like `MS Paint`