DEV Community: Loadero

Migration Of An Application Frontend To TypeScript

Loadero — Mon, 10 Oct 2022 11:36:23 +0000

We are always looking for ways to improve our products and make them more robust, secure, and maintainable. As we add more features to our products, the complexity of our code bases increases and it makes them more difficult to add or refactor the code without introducing regressions of the functionality. Since our frontend was written in plain Javascript and React, there was no way to ensure type safety of passed data between components and functions.

With the complexity of our code, we reached the point where there was no confidence in changing some parts of the application without introducing new bugs. At that point, we decided to migrate our frontend codebase into TypeScript. It is the de-facto option of choice, and no other strongly typed programming languages or static type checkers for Javascript come even close to TypeScript in terms of the size of the community, features, speed, and IDE support. Such a migration is a complex task, so we start with some preparation.

Write guidelines and plan migration

Before jumping into converting all of our code into Typescript, we created a plan. One of the important things we wanted to do first was to ensure that the style of our code is consistent. This is very important as we want to keep our code predictable, so anyone would be able to look at it and understand what’s going on there. So that’s what we did – we took a look at our existing code, put our heads together and wrote guidelines that explain standards all engineers should adhere to when they write code in TypeScript. The main points we have put in our guidelines were:

Types and interfaces
Eslint and Prettier configurations
Functional component signatures
File naming conventions
Hook definitions
Typing of Redux APIs

Once we had our new coding guidelines created, we proceeded with auditing all our files to create a strategy for the migration. We checked all imports and decided which files needed to be converted first. We wanted to convert all dependencies first like utility functions, API wrappers, hooks, configs and others, so TypeScript components don’t rely on unconverted files. Figuring out the strategy for migration didn’t take a long time as it was pretty obvious from the code what needed to be converted first.

At the time of migration, we were also shipping new features and refactoring our existing code, and all this new code was written in TypeScript. The great thing about TypeScript is that it allows us to have JS and TS files coexisting together without any issues, allowing us to implement the migration incrementally, step by step.

Setup the project

Since our project was bootstrapped with Create React App, it was very easy to configure the compiler to use TypeScript. All we had to do was add tsconfig.json file to the root of our project with some predefined rules. To improve build times and prevent issues with incompatible external library types, we enabled skipLibCheck flag in the tsconfig.json.

Once the configuration file was set up, we updated our ESlint/prettier files to use TypeScript. There is not much to be said about configuring those apart from adding some TS-related rules and installing dependencies like @typescript-eslint/eslint-plugin @typescript-eslint/parser so your prettier and ESLint work with TypeScript.

Migration

Having all the required configuration of the project done we started the migration. We have planned all required steps and proceeded with converting javascript files one by one. As we planned, we started converting files that have no dependencies on other files and gradually moved to files that are located closer to the root of the project. By following this strategy, we minimized the risk of having lots of conflicts and TypeScript errors between different files.

While doing migration we have spotted some bugs and improved the quality of our code. The majority of bugs could have been avoided if we had a type safety in place. The examples of those bugs were: missing or incorrect arguments while calling functions, accessing null values and missing props in components. All of those issues have become apparent after compiling the code into TypeScript. With type safety in place, we can define contracts by using interfaces and guarantee that components receive correct props, thus eliminating potential bugs.

One of the great improvements we have introduced to our projects was to implement type-safe translations. Since TypeScript is now able to parse key-value dictionaries in JSON files, we can determine the type of translation dictionaries. This allowed us to validate all translation keys during the compilation stage and provide some autocompletion of keys in our code editor.

Caveats

Of course, the migration didn’t go 100% hassle-free as there were a lot of changes to be done. Some parts of our codebase went much quicker, but some required more thinking and refactoring before it could be converted to TypeScript. To type our low-level API wrappers we had to restructure them a bit and heavily rely on generics so our wrappers can stay 100% reusable and work with any data they pass through. Even though we followed our plan to minimize conflicts and errors between files, we still spent quite a lot of time fixing tricky type errors while doing the migration. With complex file dependencies, you start revealing type errors of already converted files only when you start converting dependent files to TypeScript.

After putting lots of our engineer’s time and effort into the migration, we can certainly say that the quality and robustness of our frontend code are much greater now. The refactoring of our code that was already written in TypeScript has already become much easier, safer, and requires less time. It does take more time to write code in TypeScript compared to plain Javascript, but then you get that time back when you start refactoring your code.

How To Test Screen Sharing Feature In Web Applications

Loadero — Tue, 20 Sep 2022 13:36:21 +0000

Screen sharing is not just common nowadays, it is present in many different apps we use. Webinar applications allow presenters to share their screen with the audience, video calling and conferencing apps allow sharing your screen to show other attendees the information you need to share, e-learning is a lot more effective when teachers can see what’s happening with their student’s screen, the list goes on and on. Many modern applications provide the users an opportunity to share their screen, but in order to make sure the users have a great experience in the app, you have to test screen sharing features. In this blog post, we’ll explain how to do that.

How screen sharing works in web applications

In most cases, these video communication applications use WebRTC as their desired communication protocol. This protocol splits each media type into separate tracks, for example, when a participant joins the call, it’s going to have at least two corresponding tracks – one for audio, one for video. When screen-sharing is turned on, two more tracks are created – one for the audio, the other for the video. The newly created tracks aren’t any different from what is regularly made when participants enable a webcam. This means that sharing a screen is almost identical to a new participant joining the call. This allows gathering some metrics that could indicate whether the feature works, and quality was good and stable. Loadero is used a lot for testing audio and video quality in apps that use webcams and microphones, and a similar approach to WebRTC testing can be used to test screen sharing as well.

Data for all tracks are saved and can be found in the WebRTC internals dumps that we save for each Loadero participant. While analysis of the internals dump can help understand the performance, our suggestion is to check the graphs we provide first, it’s easier to understand and is less bloated.

WebRTC internals dump with over 1000 lines

One of the graphs for a single media track in Loadero.

Why should one test screen sharing feature

As described before, screen sharing is almost a de facto feature of video communication applications. But if you already tested your application manually and clicking the screen sharing button does exactly what you expected, should you include this feature in your automated load tests? Our answer is – you definitely should. Our approach to load testing and automated testing is to simulate real users’ actions as close as possible to the real world. So if you are working on an application, which allows screen sharing presentations, learning materials, etc., testing the usage of the feature will allow you to validate that screen sharing works fine under a load of high concurrent users, that the user experience and quality of audio and video does not drop when the feature is used, that the resource usage of the feature is reasonable.

How screen sharing impacts machine resource usage

In our own and our client’s tests for video calling apps, we have noticed that screen sharing creates additional strain on the available machine resources. This happens additionally to already resource-demanding environments like 1-on-1 or group video calls. We have tested this theory in the Jitsi meeting call. We ran two participant tests, both participants had 3 CPU cores and 6GB of RAM available. Both participants had their webcameras on, except one of them additionally was screen sharing. We noticed a 10%-20% increase in overall resource usage when screen sharing was enabled.

Because of this increase in resource usage, if you’re thinking about enabling screen sharing, we encourage Loadero users to look at the resources that their test use and evaluate – should I increase the compute unit value for participants? If not enough resources are available, the quality of the call can and most likely will be impacted in some way – lower resolution or lagging video. So to get usable data and insightful results from your test, make sure you assigned enough compute power to all the test participants.

How to test screen sharing

Now let’s get to the most exciting part – creating an automated test, which simulates participants joining a group call and one of the participants enabling screen sharing. Let’s go over a scenario and write a small script example for the Jitsi Meet application where we can test the screen sharing feature. Before we start, if this is your first time creating a test in Loadero, make sure to read about the basics of configuring test parameters and test participants here.

When working on a test in Loadero, all you have to do to enable screen sharing is click the button in the web UI. Similarly, as you simulate other actions in your tests, you’ll have to identify a selector to use with the click command and use the command to simulate one of the participants clicking the share screen button.

The test script we created for this example does the following:

Participants visit the URL of a Jitsi meeting room
Participants enter their nicknames and click “Join meeting” button
One of the participants (the first one in the second group) clicks the button to share the screen
1-minute delay
Each participant takes a screenshot
1-minute delay

Tip: Usually only one participant of a call is allowed to share a screen at any moment, so you will likely have to assign the action to only a single participant. To create a test script, in which one participant’s set of actions differs from the others you can follow this guide.

Here is the script we created for the test:

(client) => {
 const callDuration = 120 * 1000;
 const timeout = 30 * 1000;

 client
   .url(`https://meet.jit.si/one-on-one-${client.globals.run.id}`)
   .waitForElementVisible(".field", timeout)
   .setValue(".field", `R${client.globals.run.id}_P${client.globals.participant.id}`)
   .waitForElementVisible('[aria-label="Join meeting"]', 10 * 1000)
   .pause(2 * 1000)
   .click('[aria-label="Join meeting"]')
   .perform(() => {
       if (client.globals.group.id === 1 && client.globals.participant.id === 0) {
           client.click('[aria-label="Start / Stop sharing your screen"]')
       }

   })
   .pause(callDuration / 2)
   .takeScreenshot("WebRTCCall.png")
   .pause(callDuration / 2);
};

Important: Keep in mind, that the value might change depending on what is the set language in your browser, for the English browser version it’s “Entire screen”.

When you or a test participant clicks on the button to share screen, a pop-up to choose which screen to share will appear. Similarly, browser alerts and other pop-ups, are not intractable as elements, so handling actions to select which screen to share, is not possible from the script side, it has to be pre-selected using automation settings. If you are running your test in Loadero, the option to share the entire screen that the participant sees will be picked automatically, so you don’t need to address this.

We automatically select the “Entire screen” option on the popup

If a user shares their screen and does not navigate to another tab, a lot of times the shared media will essentially be an infinity mirror showing its own desktop in a loop. This should not impact the media quality at all, but it does create an interesting view.

If you’re testing locally, you most likely will need to add some fields to your automation settings. Currently, such a thing can be done on Chromium-based browsers using Chrome Options and Capabilities, so sadly, screen sharing does not work in Mozilla Firefox. For Google Chrome, we opted to use a launch argument that is available and is called “auto-select-desktop-capture-source”, this argument allows providing the name of the capture source which will be automatically selected when the selection window pops up.

auto-select-desktop-capture-source=Entire screen

Tip: For a better understanding and more available launch arguments for Chrome, you can check this list out.

Verifying that screen sharing works in your test

We have created a test, which includes screen sharing in the user flow, but now we have to verify that it works correctly during test runs. One of the easy ways to verify that the screen is shared is by taking a screenshot after the button was clicked. You will be able to find the screenshots in the individual participant results under the Artifacts tab.

Another similar way to make the check is by using the session recording mode. This is a handy way for debugging, that we also use a lot when working on test scripts. Similarly as with the screenshots, you will find the session recording files in the same tab and will be able to watch what the user had on the screen.

More advanced users will be able to also spot which WebRTC tracks are for the shared screen in the WebRTC internals dump or in the graphs we provide. Usually, the same user cannot show their webcam and desktop at the same time, if that is not the case for your application – it might be harder for you to set precise WebRTC assert expected values.

To automatically check if screen sharing was enabled, you can use post-run assertions. One of the assertions Loadero offers for this is the assertion for outgoing video connections. Keep in mind that if only some participants are going to enable sharing assert preconditions should be used. The expected value can change depending on the application, but the minimum usually is 2, which is going to describe the webcam and the shared screen. In some cases when joining a call there is a webcam preview, which can result in an additional “connection”.

We did go over that this media behaves the same as webcam feed, that is why the same principles apply here as well. You can read more about that in our blog post about testing communications platforms under the “WebRTC data analysis” section.

Screen sharing is one of the main features of today’s communication platforms, but is often overlooked and not tested as extensively as other features. Make sure to read other posts in this blog and take a look at our wiki, with this information you will be able properly test your product to provide a superb user experience, and it allows screen sharing – make sure to test the feature in your load tests as well. See ya on one of our next blogs, have a great testing.

Local Tests With Selenium And Python Browser Automation

Loadero — Fri, 26 Aug 2022 13:35:18 +0000

There are various reasons for running Python browser automation tests locally, the most common one for us is saving time. Loadero test runs usually take no more than 5 minutes to initialize and start execution but can reach up to 10 minutes or more depending on how busy is the test run queue. To increase the speed of development, it’s often quicker to create the test script on your local machine and then run tests on Loadero. Loadero’s test initialization time is unavoidable if you want to use one of its key features – multiple test participants (not just a few, but up to tens of thousands ). If you limit your tests to just one participant to make a quick run for checking your progress, the test can be run on your local machine, avoiding waiting times. There you can quickly develop a working script that can be run on Loadero with thousands of concurrent participants later. This blog post will show you how to set up a local test script development environment in Python for a single test participant. If you plan to do the same with Javascript, you might find this blog post about test automation with Javascript and Nightwatch framework handy.

To fully set up the script development environment, we must set up the following 2 environments:

Python environment
Browser automation environment

Python environment

The first thing you need to prepare to work on your local tests is the Python environment. In a directory of your choice create a virtual environment with

python -m venv py-script-dev-env

then activate the environment on macOS and Linux with

source py-script-dev-env/bin/activate

and on Windows with

py-script-dev-env\Scripts\activate.bat

Creating a virtual environment is not a direct necessity for local script development, but it is always a good idea to separate your Python environments to keep your global installation clean. This is what we usually do when we are working on developing a new test script.

Tip: The Python version used in this blog is 3.10.4, but using any version newer than 3.6 is also fine.

Now, we have to install the Py-TestUI framework which is used by Loadero to automate user actions in the browser.

But before doing that, one of the Py-TestUI dependencies needs to be installed separately – Selenium 4.1.0. This is because at the time of writing this blogpost the latest version of Selenium is 4.1.4, but Py-TestUI hasn’t caught up yet, so an older version of Selenium needs to be installed first. Selenium 4.1.0 is also the version that Loadero currently uses.

pip install selenium==4.1.0

Also, currently, Py-TestUI is not available on PIP (Python Package Index), but we can utilize the pip utility to install it directly from GitHub. As of writing this blog post the latest Py-TestUI version is 1.1.0. To install it run

pip install git+https://github.com/testdevlab/Py-TestUI@v1.1.0

Automation environment

Now let’s set up an automation environment that runs directly on your machine. By default, Selenium looks for a browser driver executable in the default search path. So all we have to do is install the desired browser driver and make sure that the $PATH environment variable points to it.

Browser driver is an API implementation of the W3C WebDriver standard for a specific browser. Here you can find download links for chromedriver and geckodriver. The installation process will vary between platforms.

For macOS users I suggest installing via brew, both chromedriver and geckodriver are available on it.

brew install chromedriver

brew install geckodriver

Linux users will have to download the compressed driver file from the links provided above. Extract the driver binary with

tar -xf geckodriver-v0.31.0-linux64.tar.gz

unzip chromedriver_linux64.zip

Then move the extracted binary into a directory that $PATH points to. I suggest /usr/local/bin.

mv geckodriver /usr/local/bin

mv chromedriver /usr/local/bin

Windows users also have to download the archived driver files and extract them with some utility like WinRAR. Then add the binaries to default search path. Then create a folder C:\bin by running

mkdir C:\bin

Move the driver binaries to C:\bin

Edit the PATH for your account with

setx PATH "C:\bin;%PATH%"

Finally, restart the command prompt.

To check if driver binaries are on your systems default search path run

chromedriver --version

and

geckodriver --version

Everything is working fine if the command outputs the version string of the driver you installed, that looks something like this for chromedriver

ChromeDriver 100.0.4896.60 (6a5d10861ce8de5fce22564658033b43cb7de047-refs/branch-heads/4896@{#875})

and something like this for geckodriver

geckodriver 0.31.0
The source code of this program is available from
testing/geckodriver in https://hg.mozilla.org/mozilla-central.
This program is subject to the terms of the Mozilla Public License 2.0.
You can obtain a copy of the license at https://mozilla.org/MPL/2.0/.

Everything required for running browser automation scripts in Python is installed. Let’s do a simple test that demonstrates everything working together.

Tip: You can find the basics of test automation explained and some of the commonly used commands described in our previous post about test automation with Python.

Running a test locally

Create a file in the directory that you chose as the parent directory of your environment.

touch simple_test.py

This file will contain your script. Here is a simple example script we’ll use, feel free to copy and paste it to quickly check if everything works correctly.

import time
from testui.support.appium_driver import NewDriver
driver = (
    NewDriver()
    .set_logger()
    .set_browser("chrome")
    .set_selenium_driver()
)
driver.navigate_to("https://loadero.com/")
time.sleep(10)
driver.quit()

You can find the script example in Loadero’s Github repository. To run this test, execute the following command

python simple_test.py

A Google Chrome browser window should open and navigate to Loadero’s home page and wait for 10 seconds.

Now you have a working local browser automation Python environment, but to make it feel more like a Loadero test script there are a few more steps ahead.

Using Python browser automation test script

Loadero test scripts are run as pytest fixtures. Luckily Py-TestUI already installed pytest as a dependency, so we can safely import it.

First, let’s create a new Python script that utilizes pytest.

touch test_script.py

import pytest
import time
from testui.support.appium_driver import NewDriver
from testui.support.testui_driver import TestUIDriver
@pytest.fixture(autouse=True)
def driver() -> TestUIDriver:    
    driver = (
        NewDriver()
        .set_logger()
        .set_browser("chrome")
        .set_selenium_driver()
    )

    print("starting test script execution")
    yield driver
    print("test script execution finished")
    driver.quit()
@pytest.mark.test
def test_on_loadero(driver: TestUIDriver) -> None:    
    driver.navigate_to("https://loadero.com/")
    time.sleep(10)

Here is a brief explanation of what this short script does:

@pytest.fixture(autouse=True) indicates that the function driver will be automatically used for all pytest tests. This makes it a convenient place to set up the PyTestUI driver.
yield driver pauses the execution driver function and returns the driver object. After pytest tests finish running, regardless of success or failure driver function execution will resume and the Selenium driver connection will be closed.
@pytest.mark.test gives the test_on_loadero function a special marker test that can be later referenced.
test_on_loadero contains the test script logic.

Run the script with

pytest loadero_script.py -s -m test

The -s argument indicates pytest to not capture any input or output of the test. The -m test tells pytest to run only tests marked with test. To avoid a pytest warning – PytestUnknownMarkWarning create a pytest.ini

touch pytest.ini

and predefine the test marker in that file.

[pytest]
markers =
   test: marks function as loadero script

Notice that the function test_on_loadero name matches the entry point function name in Loadero Python test scripts. Although you can name the function however you want, matching function names were chosen deliberately to showcase that the function provides an entry point for the test script both on the local setup and in Loadero. This means that the script you develop locally can be transported to Loadero by copying the test_on_loadero function and its helper functions, except the driver fixture, to a Loadero test script. The script example, its Loadero counterpart, and the pytest configuration file are available on Github.

Now your environment is complete and you can start working on your first automation script. One important thing to notice is that Loadero provides custom variables, constants, and commands for file uploads, custom network conditions, measuring command execution time, and other things available only in Loadero. You can read more about them here. These commands will not be available on your local setup.

This concludes the environment setup and you are ready to develop your Loadero scripts on your quick local setup. Once you are ready to try running the test script with multiple participants, you can do that using Loadero’s free trial. Good luck at the tests, and see you in the next blog.

Test Automation With Python

Loadero — Thu, 28 Jul 2022 05:53:47 +0000

According to IEEE Spectrum Python is the top programming language of 2021, and since April Loadero supports Python as a test script language. Many engineers agree that test automation with Python is an excellent choice, and in this blog post, we’ll show the basics of it.

Python tests in Loadero use the Py-TestUI framework. Py-TestUI wraps and implements additional features for Selenium Python binding. This means that Loadero Python tests can utilize the full power of Selenium and other cool features provided by Py-TestUI.

Test Structure

Every Python test starts with

def test_on_loadero(driver: TestUIDriver) -> None:

This function is like the main function in other programming languages. It will be called once the test execution is started. All of the test logic must originate from this function. This means that additional functions outside of the test_on_loadero function can be added, but they must be called from the test_on_loadero function to have any effect.

Driver

Loadero tests automate browser interactions as if they were done by a human to allow for repeated execution, parallel tests, and scale-up for load testing. Tests can do almost any action that a human can in a browser, for example:

Navigate to URL
Click on links and elements
Input text
Copy text

These actions are achieved in a test with the driver object. driver wraps the workhorse of the browser test – Selenium WebDriver and is of type TestUIDriver. TestUIDriver implements commonly used browser interactions to improve ease of use but at the same time allows to access the underlying Selenium WebDriver for more complex browser interactions.

driver object is initialized and configured at the start of every test and passed as an argument to the test_on_loadero function.

Let’s look at common browser interactions and how to implement them in a test script.

Navigation

To start interacting with a web page, it first has to be opened. To do that call the following method

TestUIDriver.navigate_to(url: str) -> None

Argument url is the URL of a web page to open.
Here is a simple test that demonstrates the usage of navigate_to.

def test_on_loadero(driver: TestUIDriver) -> None:
    driver.navigate_to("https://duckduckgo.com/")
    driver.save_screenshot("duckduckgo.png")

Additionally, the test calls

TestUIDriver.save_screenshot(image_name: str) -> str

This method will, as the name implies, capture the state of the browser at the moment of the save_screenshot method call and save it with the name supplied by image_name argument. After the test finishes execution, all screenshots taken during the test will be available in the artifacts tab in the Loadero result page of a single test participant.

Tip: Taking screenshots during a test is a common practice that helps test debugging or manually reviewing what was on the test participant’s screen at the important stages of your test.

Clicks

A very fundamental browser interaction is the click. To interact with an element it first has to be located with the e method.

TestUIDriver.e(locator_type: str, locator: str) -> Elements:

locator_type argument describes the location method of an element and can have values:

css – locate element by CSS selector
name – locate element by name
className – locate the element by its class name
xpath – locate element by XPath

locator – describes how to locate the element with the location method set by locator_type argument. To learn more about using CSS selectors and XPath locators in your Selenium test automation scripts make sure to read this blog post.

An attentive reader might have noticed that e method has return type Elements – plural, but I have been using elements in a singular form so far. This is because Elements object has more advanced features (for example – locating multiple elements at once, image matching) that this blog post does not cover. So in an attempt not to confuse the reader these topics will be skipped.

After locating the desired element it can be clicked on with the click method.

Elements.click() -> Elements:

There is nothing in the way for calling click right after an element has been located, but a good practice is to check if the element is visible with a runtime assertion before clicking on it. If the element is visible then it has had enough time to load and can be interacted with. Doing this allows avoiding test failures caused by attempting to click on an element that has not loaded yet. We use a lot in tests we create for our customers and advise you to do the same.

wait_until_visible method does as the name suggests – waits for the element to load and become visible before attempting further interactions.

Elements.wait_until_visible(seconds=10.0, log=True) -> Elements:

Argument seconds with a default value of 10.0 specifies the amount of time the test will wait until the element becomes visible. If the element is not visible after waiting the specified time an error will be raised and the test will fail.

Argument log with a default value True specifies whether to log the result of the wait_until_visible method call. This argument generally should not be changed and can be ignored. Many Elements methods have a log argument and it can be ignored in all of them.

Here is a simple test script showcasing everything shown so far. You can put everything learned so far to test with Loadero’s free trial. You can also find the script example in Loadero’s Github repository.

def test_on_loadero(driver: TestUIDriver) -> None:
    # Opens the search engine DuckDuckGo with default search query of
    # google.com
    driver.navigate_to("https://duckduckgo.com/?q=google.com")
    # Locates an element with a CSS selector that is pointing to the first
    # search results title, waits for the element to become visible and clicks
    # on it.
    driver.e("css", "#r1-0 > div > h2").wait_until_visible().click()
    # Ensures that the web page has loaded before taking a screenshot.
    driver.e("css", "body > div:first-of-type").wait_until_visible()
    driver.save_screenshot("google.png")

Input Text

The previous test script example was cheating a bit. Realistically a human user would insert the search query “google.com” themselves and wouldn’t be using a specially prepared URL. To simulate this behavior, tests can input text with the .send_keys method.

Elements.send_keys(value: str, log=True) -> Elements:

value argument specifies the text that will be sent to an element.
To clear the input value that was set by send_keys or an element has by default use clear method.

Elements.clear() -> None:

Retrieving Content of an element

Browser interactions can be generalized as a feedback loop where the next interaction is determined by the resulting content of a web page. Tests need a way to retrieve the content of a web page or more specifically the content of a single element to implement logic that decides what the next interaction should be. This section will cover the other half of the browser interaction feedback loop – content retrieval.

Most HTML elements have content, with few exceptions like <br> element, that is rendered to the web page and can be interacted with.

To retrieve the textual value of an element use

Elements.get_text() -> str:

Putting Everything Learned To Use

It’s time to combine all the commands we have described in a single test automation script. This script example performs a search for a specific query on the DuckDuckGo search engine and prints out the top results.

def test_on_loadero(driver: TestUIDriver) -> None:
    # The main logic of the test is in top_search_results function
    print(top_search_results(driver, "loadero", 10))
# top_search_results returns the top n search results for search_query.
def top_search_results(driver: TestUIDriver, search_query: str, n: int) -> str:
    # Limits the search result count to 10, because a single search query in
    # DuckDuckGo search engine returns only 10 results.
    n = min(n, 10)
    # Selectors.
    home_page_search_bar = "#search_form_input_homepage"
    home_page_search_button = "#search_button_homepage"
    # Opens the search engine DuckDuckGo.
    driver.navigate_to("https://duckduckgo.com/")
    # Locates search bar, waits for the element to load and sends a search query
    # to it
    driver.e("css", home_page_search_bar).wait_until_visible().send_keys(
        search_query
    )
    # Locates search button, verifies that it is visible and clicks it.
    driver.e("css", home_page_search_button).wait_until_visible().click()
    result = f"Top {n} DuckDuckGo search results for {search_query} ->\n"
    # Iterates over top n search results
    for i in range(n):
        # Creates a CSS selector that indicates to the search result title.
        selector = f"#r1-{i} > div > h2 > a.result__a"
        # Locates the search element and verifies that it is visible.
        search_result_element = driver.e("css", selector).wait_until_visible()
        # Retrives the title of search result and adds it to result string.
        title = search_result_element.get_text()
        result += f"\t {title} \n"
    return result

You can find the output of this test script in Loadero’s single participants result pages logs tab under Selenium log. The output would log something like this:

Top 10 DuckDuckGo search results for loadero ->
     Loadero - Performance and Load Testing Tool
     Loadero | LinkedIn
     Loadero Pricing, Alternatives & More 2022 - Capterra
     Loadero - Home | Facebook
     GitHub - loadero/examples: Test script examples that are ...
     Loadero - YouTube
     Loadero - your performance & load testing partner from the ...
     Application Load Testing Tools for API Endpoints with ...
     Tractors - Tractor Loader Backhoe - B26 | Kubota
     EZ Loader TMS | EZ Loader TMS

We covered the basics of working on test automation with Python. You can create a test similar to what we described and run it multiple times free of charge with Loadero’s free trial. To continue learning to use Python for your tests in Loadero make sure to visit our Wiki, where you can find information about custom commands that extend the functionality WebDriver and PyTestUI, variables that hold the relevant information about the test participant, and more script examples of test automation with Python. We will be glad to see you back again.

Testing A Communications Platform’s (CPaaS) Performance

Loadero — Tue, 12 Jul 2022 13:42:31 +0000

With the growing demand for online communication, there has been a noticeable increase in platforms to provide just that. With such service comes high expectations in quality and usability to become the best! We hear about those high expectations quite often, some people developing video calling applications that use CPaaS expect them to work and scale perfectly and don’t load test their own applications. While such an approach can cause a lot of issues in the future, it shows how important properly testing a communications platform, its video calling APIs and SDKs is. One of the best and easiest ways to notice potential issues is using a cloud testing platform to run frequent performance/load tests and gather metrics to analyze the performance, call quality and scalability.

Automated testing can reveal a variety of issues due to higher service resource demand and potential use of non-perfect network conditions. If testing a communications platform and its performance wasn’t done properly, it can result in bad user experience because of regressions, low performance, video being prioritized over audio and even unexpected behavior caused by browser differences. Tests integrated into CI/CD pipelines help our customers to always be calm and sure that their service works perfectly after every new deployment. As we see the demand for testing growing among the CPaaS and applications for video calling, we summed up some information on how Loadero helps our customers in the field to test real-time video services.

What’s important in testing a communications platform

Handling poor network connections
In real life users don’t always have a good and stable network connection when they are using your service. Some might have a high packet loss rate, others will have a slow connection or high jitter, all that can make user experience worse. And also the network conditions are not constant, they may change along the way when a user switches to another network, loses connection to his WiFi when exiting a building, has a worse connection due to entering a tunnel, etc. All this will happen every now and then. We know the importance of testing those cases and have a setting in Loadero that will allow you to check what happens when your service is used with an imperfect network connection. That’s why it is important to pick the desired network condition preset carefully when testing a communications platform – it might impact the results! You can also simulate change of the network conditions during the test run using our custom commands for that. Check out this blog post about configuring various network conditions for participants in your tests.

Quality of audio and video
Of course, regular checks of audio and video quality are necessary to keep peace of mind and know that customers get great service. To validate quality automated tests can be integrated into CI/CD pipeline and launched regularly. This can be done with small-scale tests that just measure some metrics that are the most important. But it also is a common practice to launch load tests on a regular basis to provide enough data points to make objective conclusions on system behavior. This can also be combined with competition analysis, to understand what holds your platform back from being the best in class.

To save time on analyzing such regular tests, it is a good idea to use assertions to set expectations and automatically check if they were met. We offer a wide range of WebRTC assertions to configure in your tests. You can automatically check framerate, bitrate and other metrics for incoming and outgoing data for each test participant. If you already have a WebRTC test created in Loadero, learn using asserts and explore how this can save your time and help always be aware of audio and video quality in calls on your platform.

Worldwide coverage
With many services providing video communication APIs competing, customer expectations grow constantly too. One thing that most users expect from CPaaS is low latency for all users. There are different parameters that can have an impact on latency, a very common one is location. If you can guarantee a less than 200ms latency to customers on the same continent, that’s great. But will it be the same if one user is located in Asia, another one in Europe and the third one in the US? This definitely has to be checked to confirm you are ready to serve users worldwide or to find issues to be fixed. Loadero provides access in 12 different locations across the world, allowing to create tests that will simulate real-world scenarios where users are not located in the same country or even continent. You can use the feature in your small-scale tests to compare user experience in various locations, and also you can use the same approach in your load tests to check if anything changes when the infrastructure experiences a high load. You can read more about checking performance for users in different locations in this blog post. Nowadays every service is expected to work worldwide and, of course, CPaaS are not an exception, so make sure you check the performance for users in different parts of the world.

Browser support
Not often, but in some cases, new browser versions introduce changes that can change the platform’s behavior unexpectedly. Not only that, but also bugs are fixed on each new release and new bugs are introduced as well. Having good browser coverage allows detecting differences in how your application behaves on different browser platforms. Loadero uses real browsers as test participants and we know the importance of cross-browser testing. That is why we allow configuring one of the five latest versions of Google Chrome or Mozilla Firefox for our test participants. Once a new version of the browser is released, our team works hard to get it added to our tool ASAP. A quick small-scale test that has 10 participants with 10 different browser versions can cost less than 1$ USD, but will help you make sure that your service handles the new and previous versions equally well.

Types of tests to run

Performance tests
Performance testing communications platform is where Loadero can help get insights on how well the service behaves from the user side on various configurations, including – different network conditions, locations, media and computing resources. Each of these parameters can impact the user experience in a major way, so none of those should be left untested.

WebRTC data analysis
When running tests in Loadero on applications that support WebRTC protocol, then WebRTC data will be gathered during the whole test duration and later be provided to you, these WebRTC metrics could represent how well the session went. This data is represented in graphs, data tables and also plain text data dump for each participant running the test. In the beginning without any prior knowledge WebRTC data could seem intimidating, for starters – there is a lot of it and at first sight does not give any direct information about the quality of the call. For easier use, we gather the data and provide it in aggregated statistics, these are some that might be most interesting to you: frames per second (FPS), audio/video bitrate, jitter and round trip time (RTT). In short, both – FPS and audio/video bitrate directly impact the signal quality of the call, for these metrics, generally higher is better. On the other hand jitter and round trip time might not impact the signal quality but rather overall user experience, because of stuttering, etc. For these metrics generally lower is better. So it’s important to understand how the application adapts, whether the bandwidth estimator is working correctly and quick enough. For more detailed information about all the WebRTC data fields, check out our wiki page about asserts here!

Machine statistics
In the real world each application visitor has a different computer configuration – some visitors might have an abundance of machine resources, some might be pushing the capability of their machine. When running performance tests, machine statistics are gathered by Loadero to show how much of the machine resources were used when running the test. Such data can give a good understanding of what would be the minimum requirements for stable use of the CPaaS platform, as well as quickly detect memory leaks and excessive CPU usage.

Network analysis
Digging deeper into investigating how the network behaved during the test might bring you some fresh look on how optimized is your application. That is why in the test run results we provide data about network performance. Check out the image above, you can find the same table in your test results as-well, in this table you are able to see incoming/outgoing network statistics – how many network packets were sent, got errors or were dropped and also how much network was sent in bytes. In short – when using a regular network setting there should not be any dropped packets, as well as no packet errors should appear, but obviously this can change depending on network conditions used.

There is no sure standard to determine what would be a low amount of traffic for a CPaaS, but from our experience – for start, it’s valuable to look for unexpected spikes in network packet count and bytes sent to avoid traffic congestion for the user and only then start trying to optimize the platform. The easiest way to look for such spikes is in the single participant result view in the Machine statistics tab, where you will be able to see graphs for network statistics – bytes, packets, errors, packets dropped.

Please, keep in mind that network metrics can highly change depending on what communication protocol is implemented and in some cases also the used media for the participant has a great impact in this field.

Load tests
Load testing a communications platform is very important. Imagine a scenario where a new startup launches their new live event broadcasting web application and promoters are starting to sign up to broadcast huge events, but as it turns out, a lot of these events are happening on the same weekend. How confident can the team developing this application be, that it can handle such load on the service? In this case, load testing could get extra confidence in the application and avoid potentially bad user experience for the viewers. Recently we noticed that many new live events services appear and some of them are running load tests in Loadero before an actual large event to make sure they are ready for the big day. We respect such a responsible approach and are happy to help our customers to check how their application handles hundreds and thousands of concurrent users. But if you are a provider of a communications platform service, the number of concurrent users to serve can be even greater than that. And while online event service providers usually know when the load on their service will increase, CPaaS should be always prepared for a spike in the number of users. So load testing a communications platform means checking if it is ready for success.

High application demand can cause slow service or in the worst case even downtime. For proper load tests that simulate real-life usage as closely as possible, we generate the load using the real browsers to simulate users. That is what real users do – they access a service using various browser versions which could implement WebRTC protocol differently. A load test in Loadero can have all the important parameters we mentioned earlier in it: various browser versions, different network conditions and worldwide coverage. And the best part is that if you already have a small-scale test configured for your needs, scaling the test up for large-scale load testing can be done in a matter of minutes.

Steps to start testing a communications platform

There are different ways our customers approach testing their products. Usually they prepare an application that uses the platform’s video API to be tested and create an array of tests for it. What we offer is starting with our trial and running some very basic tests free of charge. You will be able to start preparing for your larger tests and also see the full results reports to plan how you will analyze those in the future. Once that is done, you can plan test scenarios to create that will simulate real usage. This is the time to start thinking about the parameters to use in your tests and configuring them for different participants.

When you are working on your test scripts, make sure to plan how the test will work when you reuse the same script for large-scale load testing. Feel free to also contact our team if you have any questions about using Loadero for your existing or future tests, we are glad to help you use our tool effectively.

Step one – come up with a scenario
Obviously, depending on your platform specifics, the scenario can change heavily, but the main idea is clear – before any automation can be written, you need to know what should be tested. Start out by creating a scenario that includes just one or two participants. A good starting point is creating a one on one communication scenario, which will generally contain the most business-critical flow. In case group communication is available, that also is a great starting point.

Step two – write the script
This step for test automation is one of the most time-consuming and requires some knowledge about programming. Luckily we have also written a series of guides on writing Nightwatch.js test scripts, check out the first part here! At first, keep it simple and let it get advanced in time. If you can’t create a test script for your needs yourself, don’t worry. Engineers from our team can do this for you. To get the test creation cost estimates, contact us with your requirements description.

Step three – resolving issues
Let’s be honest, nobody is perfect and most likely some issues will pop up at first when developing the script. Once again, we have a separate blog post for this which covers 6 Ways of Selenium test scripts debugging, check it out – it might bring up some ideas! In short – if you have launched your script in Loadero and something is not right, here are some of the things you could do:

Join the communication yourself
Use session recording mode and then visually inspect the test execution
Add screenshots to all important actions like – joining the communication room, logging in, etc;
Carefully check Selenium log – there might be some error hiding.

Step four – adding post-run assertions
To improve the quality of testing, we suggest adding post-run assertions that will check the metrics we mentioned earlier. Obviously, you can add how many you want, but usually three to six asserts are enough at first, but when the application gets more mature, it’s not unusual to have around 20 asserts to cover most of the parameters. At start the expected values can be guessed – after some runs, you will be able to figure out the baseline. If there are known requirements for WebRTC statistics or machine resource usage statistics, then feel free to use those, so the test result immediately will be more precise. We provide almost unique 30 metric paths that you can assert against, check out our blog post to learn more about post-run assertions. Each metric path also can be used with math aggregators like – minimum, maximum, average.

Step five – scaling up
Have you done all the previous steps and everything seems to run well? If your answer is YES, then this might be a good time to start scaling up the tests. We in Loadero provide a way to scale up your existing tests in a very easy manner by using the group-participant configuration. For most scaling up is as simple as increasing the count of groups (edit the previously made one and change the count field), which in the end will increase how many participants will launch the script.

How many participants to use? That’s really up for discussion on your side, but we would suggest starting slow – don’t start with thousands of participants, most of the time it’s too much. If there is a target on how many simultaneous participants your platform would like to achieve, we suggest doing it in about 5 incremental steps, to really see how the platform is able to handle different kinds of load and how good the load is being balanced between all servers.

Properly testing a communications platform is a complex and sometimes tricky task, but if you follow the advice given in this blog post, that would be a great start. Don’t forget that you can run multiple small tests free of charge to explore Loadero’s result reports yourself and plan on your future tests.

Make sure to take a look at our wiki and visit our Blog for more articles like this. With this information you will be able to create many different insightful tests for your platform or application. Start creating your tests now by signing up for a free trial account and don’t hesitate to contact our helpful support team if you have questions.

Thanks for reading! I hope this was helpful for you!

CSS Selectors And XPath Locators In Selenium Test Automation Scripts

Loadero — Mon, 27 Jun 2022 14:05:55 +0000

One of the most important aspects of automated web application testing is having a good grasp of using locators. Locators allow retrieving DOM elements from the web page. Interacting with web elements during automated tests allows to create end-to-end tests that simulate real users behavior. In this blog post, we will talk about two types of locators – CSS selectors and XPath.

CSS selectors

CSS selectors are used to identify styled elements on a web page, since most HTML pages are styled with cascading style sheet (CSS) classes. The selector consists of a pattern that is used to find matched DOM elements in the document tree. CSS comes with an incredibly rich set of selectors, and those selectors can be mixed and matched in interesting ways.

There are 3 types of CSS selectors:

Simple CSS selector
Pseudo-selectors
Attribute CSS selectors

Simple CSS selector

The most straightforward selectors target a specific tag, id or class on a web page:

a
submit-button
.nav-bar

In CSS, classes and IDs play pretty much similar roles, but there are some differences. Both are used to target elements on the web page. However, an ID can be assigned only to a single element, when a specific class to a multiple. Using classes allows you to target a group of elements. Tags are used to create HTML elements that we can target using a tag selector too.

Some elements might appear in different places on a web page. Let’s say, we want to select an unordered list only in the footer? Well, we could do that using a combinator:.footer ul

By putting space between two selectors, we are combining two selectors in a very specific way, saying that we are only targeting an unordered list that is nested within an element with the .footer class.
The term “combinator” refers to a character that combines multiple selectors. In this case, the space character combines .footer class and ul tag to create a descendant selector. The descendant selector will apply to all descendants, no matter how deeply nested they are.

If we only want to select children of an element without targeting all descendants, we can achieve this by putting angle brackets between two (or more) selectors:.nav-bar-list > li

By using this combinator we select only direct children of the .nav-bar-list selector, even if it has another set of li elements nested deeper, so only three elements will be selected.

There are also other combinator types such as + and ~ which can be used to construct more complex queries. You read about them here.

Pseudo selectors

Pseudo selectors let us locate elements based on their current state, position within the document tree, and much more.

HTML comes with interactive elements like buttons, links, and form inputs. When we interact with one of these elements (either by clicking on it or tabbing to it), it becomes focused. It’ll capture keyboard input, so we can type into a form field or press “Enter” to follow a link. Or, let’s say you hover over a button and it changes its background or text colors. In both cases, the pseudo-classes will be applied to those elements.

To target a button that is in the hovered state, we can write our selector like this:button:hover

Pseudo selectors also allow us to select elements by using their position in the document tree. Imagine, we have an unordered list and we want to select the last list item of the list.

To achieve this we can use the pseudo selector :last-child that will select the last child of <ul> element: li:last-child

Of course, there are many other pseudo-classes available and to learn more about them you can check out this awesome resource.

Attribute CSS selectors

The attribute selectors can be used to target elements with specific attributes. The following examples will show you that this type of selector can be very flexible.

To select all elements with the specific attribute we can write our selector using a tag name followed by a pair of square brackets with the name of an attribute inside them:button[type]

This will select all buttons that have the type attribute, no matter what value that attribute holds. To target elements with attributes that hold exact values we can rewrite our selector like this:button[type="submit"]

To select elements with attribute values that contain specific words, we can put the tilde sign after the attribute name:img[alt~="year"]

If we want to find elements whose attributes begin with specified values, we can put a caret after the attribute’s name:.div[class^="nav"]

By using a dollar sign we can select elements with attributes that end with a specified value:.div[class$="bar"]

To select elements with attributes that contain a specified list of characters, we can use an asterisk sign:.div[class*="av"]

XPath

HTML and XML have a very similar structure, so XPath (XML path language) can be used to navigate through web pages written in HTML or XML. XPath uses path expressions to navigate through elements and attributes on a web page.

One of the main advantages of XPath over CSS selectors is that it allows you to traverse elements up the DOM tree, meaning that by selecting an element you can also target parent and other ancestors. Of course, with XPath, you can also go down the document tree and target children and their descendants. XPath has many built-in functions such as locating elements by text. Imagine a situation when HTML elements are being built dynamically but the text stays consistent, in that case, the only way of targeting those elements would be to find them by text. This is something you cannot do with CSS selectors.

Tip: Bear in mind that each browser engine has its own implementation, so some expressions can be inconsistent when used with different browsers.

Before jumping into syntax and expressions, let’s quickly take a look at the terminology used when working XPath:

In XPath we are working with nodes. An HTML document is made up of a tree of nodes. A node can be either an element, attribute, or text. The topmost element of the document tree is called the root element, or the root node.

Let’s image the following HTML structure:

<div class=”header”> 
    <div class=”nav-bar”>
        <ul class=”list”>
            <li class=”list-element”></list>
            <li class=”list-element”></list>
        </ul>
    </div>
</div>

When we have one element nested within another, we can say that the nested element is a child of a parent element. If a child has also nested elements, then we can say that those elements are descendants of the topmost element. Or, similarly, we can say that the topmost element is an ancestor of deeply nested elements.

In the above example, elements with class .list-element are descendants of the .header element. At the same time, the .header element is an ancestor of <li/> elements

Let’s take a look at XPath syntax for selecting nodes on a web page:

nodeName - Selects all elements with the specified node name
/ - Select elements starting from the root node
// - Selects all elements that match the specified selection
. - Selects the current element
.. - Selects the parent of the current element
:: - Selects children of the current element
@ - Selects attributes

In the following example, to select all elements that are descendants of an unordered list, you can write the following expression: //ul/li

By using two dots you can go up the document tree and select the ancestors of the list item://ul/..

Xpath also allows you to use predicates to select specific elements on a page or elements with specific values. Predicates are denoted with a pair of square brackets.

Let’s say you want to select the very first <li> element within the unordered list. To achieve this, the expression will look like this://ul/li[1]

If you need to select the last element in the unordered list, you can rewrite the expression by changing the predicate value to last()://ul/li[last()]

For selecting an element by the text node, you can use the following expression://*[text()='a']

I have shown you the examples of using selectors in the Google Chrome dev tools, but the idea is similar when you write your automation scripts. I will use the Nightwatch testing framework to write the following snippets which will navigate to some webpage, wait to ensure that an interactive element has appeared in the DOM, and finally click on it. If you want to learn using Javascript with Nightwatch framework for your automated tests, you can start with our guide to this.

CSS selector:

(browser) => {
  browser
    .url("<your url>")
    .waitForElementVisible('button[type="submit"]', 1000)
    .click('button[type="submit"]');
};

XPath Selector:

(browser) => {
  browser
    .url("<your url>")
    .useXpath()
    .waitForElementVisible('//ul/li[last()]', 1000)
    .click('//ul/li[last()]');
};

As you can see XPath selectors can be very flexible in terms of traversing the DOM and selecting elements, however complex expressions might look rather complicated and difficult to read. Of course, XPath has some more advanced features such as wildcards and several path selections that we didn’t talk about in this blog post, but having an understanding of its basic features will give you a good start in learning about this type of locators.

Conclusion

To sum up, CSS selectors are faster, easier to write, and more reliable across browsers, so they can be a perfect choice when you need to write a quick query that doesn’t require navigating up the hierarchical tree of the DOM. With the introduction of pseudo-selectors such as :nth-child, the traversal through elements with CSS has gotten much simpler. However, if you need to query an element by text or need to target ancestors of an element that will require you to go up the DOM tree, then you will be better off using Xpath.

Use CSS selectors and Xpath locators to prepare your tests for future large scale testing. Sign up for our free trial and run multiple performance tests free of charge. If you will require any assistance during exploration, make sure to contact our helpful support team.Hope you find this post useful, thanks for reading, happy testing!

A Beginner’s Guide To Test Automation With Javascript (Nightwatch.js). Part 4.

Loadero — Mon, 13 Jun 2022 05:39:59 +0000

How to make your script better in Nightwatch.js

Congratulations on getting to part 4 of the “A beginner’s guide to test automation with Javascript (Nighwatch.js)” blog series! This part will help you to improve your test script (and make it cooler). We’ll explain different style suggestions and the overall script structure that we follow in Loadero to bring your script to a new level.

Check out our previous parts to catch up:

In this blog post, you will learn the following:

Consistent code style
Element selector usage
Code organization

The code used in this article can be found in Loadero’s public GitHub examples repository here.

Prerequisites

Code editor of your choice (in Loadero we prefer Visual Studio Code).
Node.js (the latest version is preferred, in this example v14.15.0 will be used).
Google Chrome and Firefox browsers.

Consistent code style

When it comes to JavaScript code (and TypeScript), we love Airbnb’s JavaScript Style Guide and follow it almost fully. Even though it is quite a long read, it is very well written and has great examples.

We also suggest using the Prettier formatter plugin for your IDE. It follows Airbnb code style for the most part, though we have some exceptions and they are documented here in .prettierrc file:

{
    "tabWidth": 4,
    "proseWrap": "always",
    "arrowParens": "avoid",
    "trailingComma": "none",
    "singleQuote": true,
    "printWidth": 100,
    "quoteProps": "consistent"
}

Setting concrete code style guidelines and following them makes it easier to share the code in the team and reduces the time for readers to understand the code.

Arrow functions

Even though there are cases when you must use regular functions (especially if you need to use context-based this), we believe arrow functions should be used wherever it is possible. There are a number of benefits of using them and one of the biggest is their terseness.

They are ideal when using callbacks:

// Regular function
client.getText(".some-element", function (result) {
    console.log(result.value);
});
// Arrow function
client.getText(".some-element", result => console.log(result.value));

Destructuring

You may have noticed in the previous parts that we were using ({ value }) in many of our callback functions. This is another great way of making your code shorter and actually more readable.

Since Nightwatch.js passes in callback function object that in successful response case consists of sessionID, status and value properties, we can destructure them in the function parameters (check out part 3 of these series to refresh your knowledge about Nightwatch.js callbacks). When the object is destructured, its properties are extracted and passed to the function as arguments. This allows to not create new variables inside the function body

// Bad
client.getText(".some-element", (result) => {
    const value = result.value;

    console.log(value);
});

// Better
client.getText(".some-element", (result) => {
    console.log(result.value);
});

// The best
client.getText(".some-element", ({ value }) => {
    console.log(value);
});

Destructuring is a pretty powerful JavaScript feature and can be used not only for objects that are passed into functions but also for arrays and can be used for variable assignments. To learn more about it, check out this article.

Element selector usage

You might have noticed that in the previous parts we were using element selectors, e.g., #search_form_input_homepage. In a nutshell, these strings allow finding specific HTML element(s) on any page, retrieving them and doing some actions with them, for example, clicking on them or entering some text. Selectors, frankly, make the web UI automation possible – without them, there would be no precise way of interacting with site elements.

Selectors are quite a big topic to be discussed and deserve their own article. Here, we will discuss them briefly to understand their global differences.

CSS selectors

CSS selectors are generally quite succinct and often are enough to locate specifically one element. There are multiple strategies how you can do that but in Loadero we mainly use classes, IDs, attribute selectors, and pseudoselectors. This is done mainly due to their shortness.

We suggest using primarily ID because they are unique. If that’s not possible, try using classes. We also recommend using their shorthands instead of element attributes, i.e., .classname and #ID instead of [class=classname] and [id=ID].

Tip: We suggest not providing element type (div, input, etc.) and only element’s attributes. This is mainly because in many frontend frameworks like React it is simple to change element type, for example, from <p /> to <span /> unlike element attribute which most likely will require changes in the CSS file. These changes can potentially break your automation.

XPath selectors

Sometimes providing a CSS selector is not possible, e.g., when locating an element’s parent, or is way more complicated than using some XPath function such as text(). In cases like this, XPath selector can be the next option after CSS selectors.

When it comes to XPath selectors, the same rules apply to CSS selectors – try to make them as short as possible, whilst keeping readability.

The biggest disadvantage of using XPath selectors is that their engines are browser-dependent and therefore may have inconsistencies. Also, generally, XPath selectors are slower than CSS and for bigger tests may become a hurdle.

Web element ID

Web element ID locator strategy is different from CSS and XPath mainly because it is generated by the HTML rather than the website developer and the ID changes in every browsing context (and on every refresh). Nevertheless, it is used all the time in the internal workings of WebDriver.

To get a web element ID, you have to use protocol-level Nightwatch.js functions that return that element’s ID – .element() (check its documentation here). Similarly, you can use .elements() that returns an array of element IDs (check the documentation here). You still need to pass CSS or XPath selector to .element() and .elements() but you won’t have to locate the element repeatedly because you would have that element’s web element ID.

Once you have the element ID, you can use protocol action functions that accept web element ID rather than CSS or XPath selector. Nightwatch.js offers a variety of them and you can do most of the interactions that you can with CSS/XPath selectors.

Tip: Keep in mind that these functions may be flaky since there is a higher chance of getting a stale element, especially if the website is dynamic. We suggest making sure whether you really need to use protocol-level functions and perhaps regular functions are sufficient.

Code organization

Sometimes your script becomes quite big and hard to manage. For cases like these, we have come up with guidelines to increase abstraction via the usage of functions, selector objects and extracting hardcoded values into variables. This improves readability and traceability immensely. Let’s dive in, shall we?

For the following examples, we will use the script of logging into GitHub, here’s the base script.

module.exports = {
    test: client => {
        client
            .url('https://github.com/')
            .waitForElementVisible('.application-main', 10 * 1000)
            .waitForElementVisible('[href="/login"]', 10 * 1000)
            .click('[href="/login"]')
            .waitForElementVisible('#login', 10 * 1000)
            .waitForElementVisible('#login_field', 10 * 1000)
            .setValue('#login_field', 'test@example.com')
            .waitForElementVisible('#password', 10 * 1000)
            .setValue('#password', 'password123')
            .waitForElementVisible('.js-sign-in-button', 10 * 1000)
            .click('.js-sign-in-button');
    }
};

Subfunctions

Functions generally serve 2 purposes. Firstly, they serve as an abstraction layer so you can skim over the code and understand what it is doing instead of reading every line. Think of array sorting function – you don’t really care how it works, you are mainly interested in what it does on a high level and what it returns. Secondly, they reduce the need to write repetitive code. Both of these usages can be utilized in our code to simplify our script.

Let’s start by refactoring our code into 2 logical blocks, i.e., site preparation for further actions and signing in. We can do that by creating 2 functions prepare() and signIn().

module.exports = {
    test: client => {
        const prepare = () => {
            client
                .url('https://github.com/')
                .waitForElementVisible('.application-main', 10 * 1000)
                .waitForElementVisible('[href="/login"]', 10 * 1000)
                .click('[href="/login"]');
        };
        const signIn = () => {
            client
                .waitForElementVisible('#login', 10 * 1000)
                .waitForElementVisible('#login_field', 10 * 1000)
                .setValue('#login_field', 'test@example.com')
                .waitForElementVisible('#password', 10 * 1000)
                .setValue('#password', 'password123')
                .waitForElementVisible('.js-sign-in-button', 10 * 1000)
                .click('.js-sign-in-button');
        };

        prepare();
        signIn();
    }
};

Now, we can clearly see at the end of the script that this code does some kind of preparation and logins. It should be way easier to understand code functionality than reading every line like in the previous version.

As you can see there are many repetitive .waitForElementVisible().click() and .waitForElementVisible().setValue().These can be extracted into separate functions. In fact, we can also utilize default parameters because for .waitForElementVisible() we always pass 10 * 1000. Let’s do that!

module.exports = {
    test: client => {
        // Utility functions
        const waitAndClick = (selector, waitTime = 10 * 1000) => {
            client.waitForElementVisible(selector, waitTime).click(selector);
        };
        const waitAndSetValue = (selector, value, waitTime = 10 * 1000) => {
            client.waitForElementVisible(selector, waitTime).setValue(selector, value);
        };
        // Main functions
        const prepare = () => {
            client.url('https://github.com/').waitForElementVisible('.application-main', 10 * 1000);
            waitAndClick('[href="/login"]');
        };
        const signIn = () => {
            client.waitForElementVisible('#login', 10 * 1000);
            waitAndSetValue('#login_field', 'est@example.com');
            waitAndSetValue('#password', 'password123');
            waitAndClick('.js-sign-in-button');
        };
        // Main flow
        prepare();
        signIn();
    }
};

Even though the code now has become longer than in the base version, it is more traceable and less repetitive, hence easier to read and understand. This might not be as visible in such a simple script but when your script contains multiple user flows and even more actions, this becomes crucial and indeed helpful.

Selector objects

In a big script having hardcoded selectors is hard to manage. At some point, you stop understanding which selector is responsible for which element, especially if the selector is not intuitive. Another problem emerges when you use the same selector in multiple functions and the selector changes. Then you have to update all of its usages. For cases like these, we suggest grouping elements in selector objects.

Tip: Ideally, page objects should be used but since Loadero doesn’t support them and Nightwatch.js implementation doesn’t allow easy traceability, we suggest using JavaScript objects inside the test function to manage selectors. Let’s create selectors object with 2 child objects (one for prepare section, the other for signIn). This object will host all selectors used in the script. Each property will have some meaningful name to improve our abstraction (again).

module.exports = {
    test: client => {
        // Element selectors object
        const selectors = {
            landing: {
                container: '.application-main',
                signInButton: '[href="/login"]'
            },
            signIn: {
                container: '#login',
                loginInput: '#login_field',
                passwordInput: '#password',
                signInButton: '.js-sign-in-button'
            }
        };
        // Utility functions
        const waitAndClick = (selector, waitTime = 10 * 1000) => {
            client.waitForElementVisible(selector, waitTime).click(selector);
        };
        const waitAndSetValue = (selector, value, waitTime = 10 * 1000) => {
            client.waitForElementVisible(selector, waitTime).setValue(selector, value);
        };
        // Main functions
        const prepare = () => {
            client
                .url('https://github.com/')
                .waitForElementVisible(selectors.landing.container, 10 * 1000);
            waitAndClick(selectors.landing.signInButton);
        };
        const login = () => {
            client.waitForElementVisible(selectors.signIn.container, 10 * 1000);
            waitAndSetValue(selectors.signIn.loginInput, 'est@example.com');
            waitAndSetValue(selectors.signIn.passwordInput, 'password123');
            waitAndClick(selectors.signIn.signInButton);
        };
        // Main flow
        prepare();
        login();
    }
};

Let’s improve another tiny thing. We can extract selectors child objects into variables in each of the main functions using destructuring, that way we’ll have fewer properties to call when passing selectors to functions.

module.exports = {
    test: client => {
        // Element selectors boject
        const selectors = {
            landing: {
                container: '.application-main',
                signInButton: '[href="/login"]'
            },
            signIn: {
                container: '#login',
                loginInput: '#login_field',
                passwordInput: '#password',
                signInButton: '.js-sign-in-button'
            }
        };
        // Utility functions
        const waitAndClick = (selector, waitTime = 10 * 1000) => {
            client
                .waitForElementVisible(selector, waitTime)
                .click(selector);
        };
        const waitAndSetValue = (selector, value, waitTime = 10 * 1000) => {
            client
                .waitForElementVisible(selector, waitTime)
                .setValue(selector, value);
        };
        // Main functions
        const prepare = () => {
            const { landing } = selectors;
            client
                .url('https://github.com/')
                .waitForElementVisible(landing.container, 10 * 1000);
            waitAndClick(landing.signInButton);
        };
        const login = () => {
            const { signIn } = selectors;
            client.waitForElementVisible(signIn.container, 10 * 1000);
            waitAndSetValue(signIn.loginInput, 'test@example.com');
            waitAndSetValue(signIn.passwordInput, 'password123');
            waitAndClick(signIn.signInButton);
        };
        // Main flow
        prepare();
        login();
    }
};

Extracting hardcoded values into variables

Another thing that you may have noticed is that the repeated usage of 10 * 1000. What’s more, website’s URL and account credentials could be extracted as variables, hence increasing the easy of future editing. Let’s apply these suggestions!

module.exports = {
    test: client => {
        // GitHub's URL
        const url = 'https://github.com/';
        // Default timeout for `.waitForElementVisible()`
        const timeout = 10 * 1000;
        // Element selectors object
        const selectors = {
            landing: {
                container: '.application-main',
                signInButton: '[href="/login"]'
            },
            signIn: {
                container: '#login',
                loginInput: '#login_field',
                passwordInput: '#password',
                signInButton: '.js-sign-in-button'
            }
        };
        // Account credentials
        const credentials = {
            email: 'test@example.com',
            password: 'password123'
        };
        // Utility functions
        const waitAndClick = (selector, waitTime = timeout) => {
            client.waitForElementVisible(selector, waitTime).click(selector);
        };
        const waitAndSetValue = (selector, value, waitTime = timeout) => {
            client.waitForElementVisible(selector, waitTime).setValue(selector, value);
        };
        // Main functions
        const prepare = () => {
            const { landing } = selectors;
            client.url(url).waitForElementVisible(landing.container, timeout);
            waitAndClick(landing.signInButton);
        };
        const login = () => {
            const { signIn } = selectors;
            client.waitForElementVisible(signIn.container, timeout);
            waitAndSetValue(signIn.loginInput, credentials.email);
            waitAndSetValue(signIn.passwordInput, credentials.password);
            waitAndClick(signIn.signInButton);
        };
        // Main flow
        prepare();
        login();
    }
};

The final script is available in our GitHub repository here.

Summary

Today you’ve learned how to rewrite your automation script to make it more efficient, readable and most importantly – scale well for larger tests. Remember, that these are only our suggestions that are based on our current experiences. You can always adapt them to fit your project requirements and personal preferences.

We hope that you enjoyed reading this series of blog posts and learned something useful. If you have any comments or suggestions, share them with us, we are very curious to know your opinion. Happy testing!

A Beginner’s Guide To Test Automation With Javascript(Nightwatch.js). Part 3.

Loadero — Tue, 07 Jun 2022 05:58:54 +0000

Callback functions and command queue in Nightwatch.js

Welcome to the “A beginners guide to test automation with Javascript(Nightwatch.js)” blog series part 3! If you have missed out on the previous parts, make sure to read part 1: Introduction to Nightwatch.js and part 2: The most useful Nightwatch.js commands in our blog.

In this article we will look at callback functions and command queue in Nightwatch.js, and as always – feel free to skip to any part you are the most interested in.

Callbacks
Command queue

Code used in this article can be found in Loadero’s public GitHub examples repository here.

Prerequisites

It is really recommended that you have read part 1 and part 2 of our “Beginner’s guide to test automation with Javascript(Nightwatch.js).”
Text editor of your choice (in Loadero we prefer Visual Studio Code).
Node.js (the latest version is preferable, in this example v14.15.0 will be used).
Google Chrome and Firefox browsers.

Callbacks

General

Nightwatch.js is a framework that utilizes the concept of a callback function (or shortly “callback”) heavily. But what is a callback function exactly?

A callback function is a function passed into another function as an argument, which is then invoked inside the outer function to complete some kind of routine or action.

(From MDN Web Docs)

For example, a callback can be as simple as the following code.

const outerFunction = callback => {
    const callbackResult = callback();
    console.log(callbackResult);
};
const callbackFunction = () => {
    return "Callbacks are cool!";
};
outerFunction(callbackFunction); // => "Callbacks are cool!"

outerFunction() has one parameter defined, namely, callback. callbackFunction definition is passed as an argument to outerFunction() which is then called inside the outer function body. Then the callback’s result is saved and logged in the console.

Important: Notice that callbackFunction is passed as an argument without function call parenthesis, i.e., (). That is done because a function definition is being passed rather the result of the function call.

Nightwatch.js callbacks

Since Nightwatch.js commands are chained one after another, they can’t return any value that can be assigned to a variable. To solve this problem, Nightwatch.js offers to utilize the “returned” value in a callback function, where it can be processed. This callback should accept either no arguments if you don’t want to process the returned value or accepts 1 parameter of type Object.

Let’s take a closer look at callbacks on .getText() Nightwatch.js function (check its documentation here).

To see the contents of the response object, let’s write the following code.

module.exports = {
    test: client => {
        client
            .url('http://example.com/')
            .waitForElementVisible('div', 10 * 1000)
            .getText('h1', response => console.log(response));
    }
};

// Logs out the following object
{
     sessionId: '8f3d666f316e6aa063b6ed70cfeaafff',
     status: 0,
     value: 'Example Domain'
}

As you can see in the callback function response object was logged out with its 3 properties:

sessionID- ID of the particular Nightwatch.js session. Generally, it is not that useful but can be helpful when you are running tests locally in a concurrent manner.
status– indicates the status of the request. Simply put: 0 is a successful request and anything else indicates that an error has occurred.
value– returned value of the command, in this case, the text of HTML element is returned. In most cases, you will be interested in this field since it can be used later in the callback. For example, you can assert whether the text matches your expectations or just save it in some variable for future processing.

Tip: You can name callback function parameter as you like but response or result are used the most often.

Now let’s take a look at what happens if the command fails to execute properly.

module.exports = {
    test: client => {
        client
            .url('http://example.com/')
            .waitForElementVisible('div', 10 * 1000)
            .getText('non.existing.element', response => console.log(response));
    }
};

// Logs out the following object
{
    status: -1,
    value: {
        error: 'An error occurred while running .getText() command on <non.existing.element>: ',
        message: 'An error occurred while running .getText() command on <non.existing.element>: ',
        stack: 'NoSuchElementError: An error occurred while running .getText() command on <non.existing.element>: \n' +
        '    at GetText.noSuchElementError (/Users/loadero/loadero/test/test-scripts/scripts/javascript/node_modules/nightwatch/lib/element/command.js:399:12)\n' +
        '    at Object.errorHandler (/Users/loadero/loadero/test/test-scripts/scripts/javascript/node_modules/nightwatch/lib/api/element-commands/_baseElementCommand.js:42:30)\n' +
        '    at PeriodicPromise.runAction (/Users/loadero/loadero/test/test-scripts/scripts/javascript/node_modules/nightwatch/lib/utils/periodic-promise.js:64:29)\n' +
        '    at processTicksAndRejections (internal/process/task_queues.js:93:5)'
    }
}

Oh, that looks differently, doesn’t it? The 'response' object fields look changed, for example, 'sessionID' is not present anymore, so let’s see what fields are present:

status– this field doesn’t have 0 value since this request is not successful anymore.
value– this field now doesn’t have string value as in the previous example but rather a whole object.
value.error– the error that was raised when running the previous command.
value.message– a more detailed explanation for the value.error. In this case, they are the same but in different scenarios, this field may host a more detailed explanation.
value.stack– a stacktrace where this error occurred (notice that Nightwatch.js logs specific line and column of the file where the specific error was triggered to help you debug).

Now, when you know what are callbacks and how they are used in Nightwatch.js, let’s look at the command queue!

Command queue

Now buckle up because this is the most challenging concept of Nightwatch.js to grasp but don’t be scared away by this – after reading this section, you will understand it well!

Before we get started, let’s take a look at the following example

module.exports = {
    test: client => {
        let text;
        client
            .url('https://duckduckgo.com/')
            .perform(() => {
                text = 'first'; // assigns 'first' to text
            })
            .setValue('#search_form_input_homepage', text)
            .getValue('#search_form_input_homepage', ({ value }) => console.log(value))
            .clearValue('#search_form_input_homepage') // clears input field
            .perform(() => client.setValue('#search_form_input_homepage', text))
            .getValue('#search_form_input_homepage', ({ value }) => console.log(value));
        console.log('second');
    }
};

And when this script is run, then the following is logged in the console

second
undefined
first

Some of you may get confused, to say the least, by this point. Why Nightwatch.js doesn’t log values like this as they should be read – from top to bottom like this?

first
first 
second

The reason for this behavior is thoroughly explained in Understanding the Command Queue Nightwatch.js GitHub wiki (it is a long read but it’s worth it). In this section, we will not look in such detail, this is going to be a summary of sorts. By the way, this command queue example can be found in Loadero’s public GitHub examples repository here.

TL; DR
Nightwatch.js firstly schedules commands in the queue, before executing them. All variables used in scheduled commands are set to their initial values.

Javascript is always executed first, before executing scheduled Nightwatch.js commands.

Then the commands are executed and run in a FIFO (first-in, first-out) manner.

If the command has a callback, then the callback is scheduled right after the outer function.

Once the outer function finishes its execution, the callback is called.

Command queue creation
Nightwatch.js utilizes the queue concept for executing various functions. Since that is a queue it follows FIFO rule, so the first command you call is run the first, then the second and so on. This is a key point to remember because Nightwatch.js doesn’t execute the functions as they are called, instead it schedules them in a queue (which essentially is the reason why commands are not executed as you would expect them to be run).

Let’s take a look at the following example

module.exports = {
    test: client => {
        client
            .url('http://example.com/')
            .click('a');
    }
  };

When Nightwatch.js runs the test, it calls test() function which then calls each of these 3 commands. Since they are synchronous, each of these commands is added to the queue one after another. This results in the following queue.

[
  {command: 'url', args: ['http://example.com/']},
  {command: 'click', args: ['a']}
]

Only then Nightwatch.js traverses the queue and calls each command separately.

Callbacks in the command queue
Let’s add a callback to .click() function and see how it changes the queue.

// test with 'click button' test case
module.exports = {
    test: client => {
        client
            .url('http://example.com/')
            .click('a', () => client.pause(5 * 1000))
    }
};

Initially, the command queue looks just like before

[
  {command: 'url', args: ['http://example.com/']},
  {command: 'click', args: ['a', callback]}
]

When Nightwatch.js begins to traverse the queue, it executes .url() command, waits for it to finish, only then executes .click() with the provided arguments, and then calls the callback. When the callback is called, it is added to the queue right after .click() so it can be executed as the next function. This results in the following queue:

[
  {command: 'url', args: ['http://example.com/']}, // executed
  {command: 'click', args: [a, callback]}, // executed
  {command: 'pause', args: [5000]}, // <-- just added
]

Important: One of the most important aspects of the Nightwatch.js command queue is that commands are not executed at the moment they are scheduled. But rather commands get scheduled, then JavaScript code is executed and only then Nightwatch.js scheduled commands get executed one-by-one in queue’s order. This is the reason why console.log('second') in the example before was executed first and only afterward console logs in Nightwatch commands.

Variable resolution in the queue

When it comes to variables in Nightwatch.js, you always have to take into consideration the command queue. If you don’t, then variables might have uninitialized values or just have them incorrectly set.

Initial variable values
Take a look at the following code snippet from the already mentioned example.

module.exports = {
    test: client => {
        let text;
        client
            .url('https://duckduckgo.com/')
            .perform(() => {
                text = 'first'; // assigns 'first' to text
            })
            .setValue('#search_form_input_homepage', text)
            .getValue('#search_form_input_homepage', ({ value }) => console.log(value))
        // ...
    }
};

In this example, console.log(value) logs undefined. The reason for this is simply that when Nightwatch.js commands queued, their arguments are assigned with the values of that moment. In this case, JavaScript assigns undefined to text because the variable is not initialized (only declared) and, when .setValue() is called, Nightwatch.js uses initial text value instead of the one assigned in the .perform() callback.

Variables inside callbacks
Now, let’s take a look at why variables inside callbacks have proper values, like in this example.

module.exports = {
    test: client => {
        let text;
        client
            .url('https://duckduckgo.com/')
            .perform(() => {
                text = 'first'; // assigns 'first' to text
            })            
            .perform(() => client.setValue('#search_form_input_homepage', text))
            .getValue('#search_form_input_homepage', ({ value }) => console.log(value));
        // ...
    }
};

Here console.log(value) logs out 'first'. The reason for such behavior is that when Nightwatch.js adds callbacks to the queue, it adds them with values of that current moment, so when they are executed, they have “the latest” values. In this case, when .perform() callback is added to the queue, it has text with a value of 'first'.

The magic of .perform()
At the beginning of this blog post, it was mentioned that .perform() with no parameters is mainly used for dealing with command queue execution order. The reason for that is simply because it allows wrapping some other functions inside a callback, hence making them use the latest variable values. Like so

client.perform(() => client.setValue('#search_form_input_homepage', text));

Tip: It is really recommended to use arrow functions for callbacks, mainly because of their conciseness. The recommended code style will be looked at in more detail in the next part of this series.

Another important reason why .perform() is often used, is to get rid of the callback hell where inside a callback you have a callback that has a callback inside that…, like this

module.exports = {
    test: client => {
        let text, text2, text3; // ...
        client.getValue('#input', response => {
            text = response.value;
            client.getValue('#' + text, response => {
                text2 = response.value;
                client.getValue('#' + text2, response => {
                    text3 = response.value;
                    client.getValue('#' + text3, response => {
                        // ...
                    });
                });
            });
        });
    }
};

To avoid such situations, wrap these functions inside multiple .perform() calls, like this to make your code easier to follow and increase its readability.

module.exports = {
    test: client => {
        let text, text2, text3; // ...
        client
            .getValue('#input', response => {
                text = response.value;
            })
            .perform(() =>
                client.getValue('#' + text, response => {
                    text2 = response.value;
                })
            )
            .perform(() =>
                client.getValue('#' + text2, response => {
                    text3 = response.value;
                })
            )

            // ...
    }
};

Summary

Today you’ve learned what is a callback and how to handle Nightwatch.js responses inside the callback itself. You also learned the challenging Nightwatch.js command queue and its unusual execution order of commands. If you read previous parts of these series, you should be able to create quite complex test scripts and run them locally testing or launching them at a scale in Loadero.

See you in the next and final part of “A beginner’s guide to test automation with Javascript(Nightwatch.js)” series that will contain some of the best practices used by our engineers to write better scripts, so don’t miss it. Hope this helps you. Thank you for reading, and let's test together!

A Beginner’s Guide To Test Automation With Javascript (Nightwatch.js). Part 2.

Loadero — Fri, 09 Apr 2021 07:08:56 +0000

How to write a script in Nightwatch.js

Welcome to the “A beginners guide to test automation with Javascript(Nightwatch.js)” blog series part 2! If you have missed out on the first part, you can read it here.

In this article we will look into the following and as always – feel free to skip to any part you are the most interested in:

The most useful Nightwatch.js commands:
.url()
.waitForElement…()
.click()
.setValue()
.pause()
.saveScreenshot() / .takeScreenshot()
.perform()
The final script

Code used in this article can be found in Loadero’s public GitHub examples repository here.

Prerequisites

It is really recommended that you read “A beginner’s guide to test automation with Javascript(Nightwatch.js). Part 1.” so you know how to set up Nightwatch.js locally.
Text editor of your choice (in Loadero we prefer Visual Studio Code).
Node.js (the latest stable version is preferable, in this example v14.15.0 will be used).
Google Chrome and Firefox browsers.

The most useful Nightwatch.js commands

Nightwatch.js framework has tons of functions at your disposal that are really well-documented (check its documentation for yourself). These functions range from basic user interactions such as clicks and inputs to more sophisticated ones such as changing the browser window’s size or setting cookies. They all come in handy once in a while but there are some that will be used pretty much all the time. Let’s take a look at them, shall we?

`.url()`

As you might have already noticed, this tiny function is usually at the beginning of any script. The reason for that is simple – it opens the desired website and, without calling it, you wouldn’t be able to automate other actions.

I might add that this function has a second use. It can retrieve the current website’s URL via a callback (check out the example below). To learn more about .url(), look into Nightwatch.js documentation here.

// Example usage of .url()
// Opens specified website
client.url('https://loadero.com/home');
// Retrieves current website’s URL
client.url(({ value }) => console.log(value)); // => https://loadero.com/home

P.S. What exactly that ({ value }) means you will learn in part 4 of this blog series but at the moment you can check MDN docs on object restructuring if you want to learn more.

`.waitForElement…()`

Even though in the previous part .waitForElementVisible() has been looked at, there is more to this command. First of all, .waitForElementVisible() is not the only command that waits until the element is at some state because visible is not the only possible element’s state. An HTML element can have any of the following states:

Present – element is present in HTML DOM.
Visible – element is visible for the end user. If you want to look into what defines the element being visible, we recommend checking out WebDriver’s documentation on element displayedness.

Nightwatch.js enables you to wait until the element is (not) present or visible using any of the following functions:

.waitForElementVisible()
.waitForElementNotVisible()
.waitForElementPresent()
.waitForElementNotPresent()

Each function must have only the element selector (uses CSS selectors by default) passed as an argument. All other arguments are optional (you can check available parameters, for example, for .waitForElementVisible() function here) but we really recommend explicitly passing timeout, which by default is 5 seconds as per default configuration. This parameter defines the time after which the function should fail if the element fails to meet the expected state. For example, if you use .waitForElementVisible('some.selector', 10 * 1000) and the element is not visible within 10 seconds, the test stops its execution because the assertion failed.

This function is usually used to validate whether the element has reached the specified state. For example, once the page is opened, it is recommended that it is checked if the main container is loaded before interacting with the further elements, that way you make sure that the website is actually loaded. Another use case is when an element is checked to be visible before clicking on it.

// Example usage of .waitForElementVisible()
// Without timeout argument (by default it is 5 seconds)
client.waitForElementVisible('.main-container');
// With timeout argument
client.waitForElementVisible('.main-container', 10 * 1000);

`.click()`

This function is one of the most simple functions in Nightwatch.js. You only have to pass the element’s selector you want to click on. In general, we recommend calling .waitForElementVisible() beforehand. Yes, exactly ...Visible. That way you assure that the element is actually visible, and, most importantly, interactable so the click command successfully executes. To learn more about this command check out its documentation here.

// Example usage of .click()
client
    .waitForElementVisible('.some-element')
    .click('.some-element);

Tip: Websites often don’t correctly update the element currently in focus. For instance, when clicking on the submit button, the form isn’t submitted. This usually happens because the input form was focused and this click only removed the focus from it and didn’t change the focus to the clicked button. In such cases, the element, i.e., the button, has to be clicked twice, otherwise, the desired functionality will not be triggered. Before double-clicking all elements, check if that’s your case.

`.setValue()`

Usually, users have to enter some text themselves, be it a search input box, registration form or just some modal with input fields. This function has 2 required fields: a selector and input value. To learn more about this command, check out Nightwatch documentation for it here.

// Example usage of .setValue()
// Enters “john.doe@example.com” into the field and sends ENTER keypress
client
    .setValue('.trial input', 'john.doe@example.com')
    .setValue('.trial input', client.Keys.ENTER);

Tip: client.Keys is a map consisting of various UTF-8 characters that are usually used for imitating user keypresses, e.g., ESCAPE or ENTER. Most of the WebDriver specified keys are implemented in Nightwatch.js and can be used from the list here.

`.pause()`

The .pause() function does literally what it claims to do – it suspends the script execution for the specified time.

In the previous blog post we have looked at pauses only as a means for manually validating the script’s execution. That is the most common use case for pauses.

Important: Using JavaScript’s setTimeout() will produce unexpected and inconsistent results due to Nightwatch.js command queue which will be explained in the next part.

Another use for this command is to generate data. Yes, stopping script execution does not necessarily mean not doing anything. For instance, when having a video and audio call with multiple participants using WebRTC protocol, the end user is not actively navigating the website but rather providing input for the camera and microphone. This can be easily simulated by having a pause in the script (hence not navigating the website) and providing a fake audio and video input. During that pause, the participant will continue having a call and will be generating various WebRTC statistics.

Important: When testing WebRTC solutions, always add a pause, for at least 1 minute, to gather the necessary metrics to analyze them later. This data will be collected in WebRTC internals dump that has many metrics that can help understanding potential problems for the application under test. Check out this article to learn more about WebRTC solution automated testing.

The only argument that has to be passed for .pause() is the pause time in milliseconds. More info about .pause() can be found here.

// Example usage of .pause()
client.pause(5 * 1000);

`.saveScreenshot()` / `.takeScreenshot()`

Either you use the original Nightwatch.js command .saveScreenshot() or Loadero’s custom command .takeScreenshot(), they essentially do the same thing – take a screenshot of the current view.

The difference is that by using .takeScreenshot() in the Loadero script, the screenshot will be available in the test run artifacts. Also, .takeScreenshot() allows passing exitOnFail parameter, which will stop test execution if an error occurs during the command’s execution. To learn more about this custom command, check out its documentation here.

// example usage of .takeScreenshot()
client.takeScreenshot('screenshot.png');

`.perform()`

To be frank, this is one of the most confusing commands in the whole Nightwatch.js framework but bear with me – it will make sense. This function allows passing a callback function as an argument that will be executed before calling the next Nightwatch.js function. .perform() documentation can be found here. What’s more, this callback function has 3 distinct flavors, a.k.a. options:

1) No parameters – only a callback function has to be passed. It is run right away without waiting for its execution to end before calling the next Nightwatch.js command. This comes in handy when you have to work with the command queue which will be looked at in the next part of this series.

// example usage of .perform(() => {})
client.perform(() => someCallbackFunction());

2) One parameter (done) – allows asynchronous execution of the callback by providing a done() callback function to indicate that the callback has finished running. This usually is used for executing functions that must be executed before proceeding, e.g., retrieve data from some API endpoint or establish a connection with the database. Because Nightwatch.js won’t wait until the callback function has finished its execution before calling the next command, done() function has to be called to indicate the end of the execution. This behavior is similar to JavaScript’s Promise resolve()/reject().

// Example usage of .perform(done => {})
client
    .perform(done => {
        retrieveDataFromDatabase();
        done();
    })

Important: Nightwatch.js has a default internal timeout when done() is used. If the function doesn’t complete within 10 seconds, then the script fails. To avoid such inconvenience, Loadero created .performTimed() custom command (check it out here) that allows overriding default timeout and works just like .perform().

3) Two parameters (api , done) – allows asynchronous execution with Nightwatch API object passed in as the first argument (this object is the same as client we have used so far) and done callback function as the second argument. This is rarely used for regular web UI automation. This api object is mostly useful when creating custom commands but we will not look into those since that is out of this article’s scope.

// Example usage of .perform((api, done) => {})
client.perform((api, done) => {
    api.waitForElementVisible('.someElement', 10 * 1000);
    someFunction();
    done();
});

The final script##

All these Nightwatch.js commands can be put to use in the following scenario:

Open loadero.com homepage and wait for its main container (.home) to load.
Wait for the cookie banner (.accept) to be visible and accept it.
Enter “john.doe@example.com” in the trial start form (.trial input).
Wait until the registration view loads using .pause().
Take a screenshot of the result we have so far.
Log in console that the script has finished its execution using .perform(done => {}).

module.exports = {
    test: client => {
        client
            .url('https://loadero.com/home')
            .waitForElementVisible('.home', 10 * 1000)
            .waitForElementVisible('.accept', 10 * 1000)
            .click('.accept')
            .setValue('.trial input', 'john.doe@example.com')
            .setValue('.trial input', client.Keys.ENTER)
            .pause(10 * 1000)
            .takeScreenshot('screenshot.png')
            .perform(done => {
                console.log('The script has finished its execution');
                done();
            });
    }
};

The code for this final script can be found in Loadero’s public GitHub repository.

Summary

Today you’ve learned the most common Nightwatch.js commands and various uses for each of them. By knowing that almost every command allows passing a callback as a parameter, now you know how to handle Nightwatch.js responses inside the callback itself.

In case you have any questions or want to learn more Nightwatch.js commands, we recommend checking out Nightwatch.js official documentation and their GitHub page. Don’t forget to apply to Loadero‘s free trial to run your code on a cloud platform in multiple locations with various configurations.

How To Create An Automated Login Page Test And Use Multiple Credentials

Loadero — Wed, 03 Mar 2021 09:21:25 +0000

Countless things should be tested within web applications but among them, one of the most important scenarios to be tested and even load tested is the user login process. User login serves as a medium between providing users with most of the product features and therefore requires careful and thorough testing. If your website allows users to log in, a login page test is necessary. An automated test for this can be easily created for this. In addition you can scale it up to validate that the login process works perfectly under a high load too.

When the product reaches a certain user base it is highly recommended to load test the service. A lot of different issues can pop-out regarding usability, performance, and service stability. This case is extremely relevant for services that can see bursts of logins in short timeframes, such as ticket resellers starting to sell tickets to a concert. Leaving this scenario untested can negatively impact service success. In other words, being unable to log in can deter potential customers from choosing your service in favor of a different one. In this blog post, we will be focusing on designing a load test for login functionality. We will also show two ways of using multiple credentials in the test. We will use Loadero, a SaaS for could-hosted end-to-end testing. You can sign up for our free trial to explore features and see if our service fits your needs. So let's begin.

Login test data preparation

For the actual test, we will need to have login credentials containing email and password. These credentials will be provided for the web application sign-in form. As such there are multiple ways how we can make use of them. We will be taking a look at 2 possible scenarios of saving and utilizing them. Examples in this blog post will be written using Nightwatch+Javascript programming language, but all of these examples can easily be replicated using TestUI+Java language as well. If you are new to using Nightwatch, here is a beginner’s guide on writing a test script in Nightwatch+Javascript.

Using hardcoded credentials

First of all, we can hardcode some user emails and passwords at the top of the test script for easier manipulation. For this, we will create an array of containing objects – each consisting of email and password fields. Each object can be counted as a single login credential pair. The downside of using this approach is that these credentials are not located in a secured place, but are rather hardcoded in the script and can easily be accessed by other project members.

client => {
    // Define credentials for 3 different participants.
    const credentials = [
        {email: 'example1@loadero.com', password: 'password123'},
        {email: 'mock@google.com', password: 'TestPassword1'},
        {email: 'laugh@yahoo.com', password: '1PasswordTest'}
    ];
    // Retrieve single participant credentials.
    const partCredentials = credentials[client.globals.participant.id];
    // Rest of the test script will go here.
}

Using externally hosted credentials

There is a possibility when single credentials will not be enough and we would like to have, for example, 100 dynamic unique credential pairs for your login page test. The most common way to handle this is to host credentials externally and retrieve them using API requests. The downside of this method is that credentials need to be hosted somewhere, like in the AWS S3 bucket. After retrieving necessary credentials we can map them in an array of objects where each object would contain email and password. This also means that API should return JSON response in the format similar to the one seen in the first example, e.g., an array of objects with email and password keys.

When these credentials are retrieved and saved into a variable within the script, in Loadero we can retrieve a pair using unique internal Loadero participant IDs (NightWatch+Javascript and TestUI+Java) and pass them into the sign-in form. In general, the response format can be arbitrary – list of objects, list of usernames with a common password, etc. Example JSON response:

[
    {
        "email": "example1@loadero.com",
        "password": "password123"
    },
    {
        "email": "mock@google.com",
        "password": "TestPassword1"
    },
    {
        "email": "laugh@yahoo.com",
        "password": "1PasswordTest"
    }
]

Tip: We recommend retrieving these credentials in a secure way, e.g., providing a one-time access token in the request header. This ensures that your data won’t be easily captured and ensures your or your client’s security. To do so, see your external service provider documentation.

client =>  {
    // Define a variable which will contain participant credentials.
    let participantCredentials = {};
    const loadCredentials = (client, done) => {
        // Calling request to retrieve data from https://example.com/credentials url
        request(
            { url: 'https://example.com/credentials' },
            (error, response, body) => {
                // If error was encountered this will stop test execution for particular participant.
                if (error) throw new Error(error);
                // Parsing received data into our credentials variable.
                const credentials = JSON.parse(body);
                // Retrieving single participant credentials based on unique internal Loadero variables.
                participantCredentials = credentials[client.globals.participant.id];
                // Calling done callback function signaling that this function has finished retrieving data and test script can continue.
                done();
            }
        );
    }
    // Requesting credentials from external source, parsing response and saving participant credentials variable.
    client.perform(done => loadCredentials(client, done));
    // Now we can access credentials for this participant using `participantCredentials.email` and `participantCredentials.password`
}

Loadero test and participant setup

Now that we have gotten basic data preparation covered we can move on to setting up a login page test within Loadero. If you haven't done this before, here is a step-by-step guide to creating a test. For simplicity we will use the hardcoded credentials approach in this example. You can create a similar test using externally hosted credentials as well, the difference will be only in the script. To start it all up go to one of your account prepared projects and press the New Test button.

We will be creating a test using these parameters:

Title: Login test
Test mode: Performance test
Increment strategy: Linear participant
Start interval: 1s
Participant timeout: 1min
Script:

client => {
    // Define login page URL to be used in the script later on.
    const logInPage = 'https://loadero.com/login';
    // Define credentials for 3 different participants.
    const credentials = [
        {email: 'example1@loadero.com', password: 'password123'},
        {email: 'mock@google.com', password: 'TestPassword1'},
        {email: 'laugh@yahoo.com', password: '1PasswordTest'}
    ];
    // Define max acceptable wait time for element to become visible.
    const waitTimeout = 10 * 1000;
    // Select participant credentials from the hardcoded credential list.
    const partCredentials = credentials[client.globals.participant.id];

    client
        // Navigate to the login page.
        .url(logInPage)
        // Wait for the page body to load in.
        .waitForElementVisible('body', waitTimeout)
        // Take a screenshot of the sign-in form.
        .takeScreenshot('logInPage.png')
        // Fill in the email field in the sign-in form.
        .setValue('#username', partCredentials.email)
        // Fill in the password field in the sign-in form.
        .setValue('#password', partCredentials.password)
        // Take a screenshot with filled in sign-in form fields.
        .takeScreenshot('formFilled.png')
        // Click the login button.
        .click('button[type="submit"]')
        // Wait until the projects page has been loaded after user authentication.
        .waitForElementVisible('.projects', waitTimeout)
        // Take a screenshot of the authenticated user on the projects page.
        .takeScreenshot('loggedIn.png');
}

Now, that we have created this test, let’s create 1 participant group for this example. Create it with the title set to “Test group” and count set to 1.

We will create one participant in the group and set its count to 3. So we will have three test participants with identical configurations. Each of them will use different credentials in the test run.

Participant parameters:

Title: Participant
Count: 3
Compute units: G1
Browser: Latest Google Chrome
Location: US West – Oregon
Network conditions: Default network settings
Media: Built-in Video + Audio feed Now we have everything ready to run a test to validate login process functionality. The test will take screenshots in 3 phases of the test: opened login form, login form with entered credentials, and projects view once the user has logged in. For more in-depth test report explanations you can check out this blog post. ##Load testing login process As mentioned before there are cases where you should test a login page in increased load capacity. For these cases, we would suggest using the 2nd way of preparing data for the test. As for the process itself, it would be some simple adjustments. In the previously described example the test participants finish the test once the page is loaded after logging in and a screenshot is taken. To generate load on the login process we need to create a loop where the test participant opens the login page, does the authentication, and then just logs out of the service. The test run will continue until all the participants perform these actions for a set number of times.

Important: Always limit how many times the loop will be executed, otherwise it may result in an infinite loop, which may result in participant timeout and unpredictable load on the website.

Here is an example script for a login process load test in which every participant will log in and log out for 15 times:

client => {
    // Initialize variable for maximum wait time spent loading page/element
    const loadTimeout = 30 * 1000;
    // Number of times to navigate through the page
    let iterations = 15;
    // Define login page URL to be used in the script later on.
    const logInPage = 'https://loadero.com/login';
    // Define logout page URL to be used in the script later on.
    const logOutPage = 'https://loadero.com/logout';
    // Define a variable which will contain participant credentials.
    let participantCredentials = {};
    const loadCredentials = (client, done) => {
        // Calling request to retrieve data from https://example.com/credentials url
        request(
            { url: 'https://example.com/credentials' },
            (error, response, body) => {
                // If error was encountered this will stop test execution for particular participant.
                if (error) throw new Error(error);
                // Parsing received data into our credentials variable.
                const credentials = JSON.parse(body);
                // Retrieving single participant credentials based on unique internal Loadero variables.
                participantCredentials = credentials[client.globals.participant.id];
                // Calling done callback function signaling that this function has finished retrieving data and test script can continue.
                done();
            }
        );
    }
    // Requesting credentials from external source, parsing response and saving participant credentials variable.
    client.perform((client, done) => loadCredentials(client, done));

    client.perform(() => {
        // Loops while specified iteration count has been reached
        while (iterations) {
            client
                // Navigate to the login page.
                .url(logInPage)
                // Wait for the page body to load in.
                .waitForElementVisible('body', loadTimeout)
                // Fill in the email field in the sign-in form.
                .setValue('#username', participantCredentials.email)
                // Fill in the password field in the sign-in form.
                .setValue('#password', participantCredentials.password)
                // Click login button.
                .click('button[type="submit"]')
                // Wait until the projects page has been loaded after user authentication.
                .waitForElementVisible('.projects', loadTimeout)
                // Navigate to the logout page.
                .url(logOutPage)
                // Wait for the page body to load in.
                .waitForElementVisible('body', loadTimeout);

            // Reduces remaining iteration count by 1
            iterations--;
        }
    });
}

We have to update the test to load test mode and alter the start interval and participant timeout fields accordingly. In this example we will have 50 participants, the start interval of 50 seconds, and participant timeout of 5 minutes. This means 1 participant per second will be joining.

One more thing that we need to adjust is the participant count. To change that up, just open the test participant page and change the participant count field to a desirable result. You can read more about transforming your performance test to a load test here. 100 compute units included in the Loadero free trial plan, you can launch a test like we have in the example twice free of charge.

Conclusions

If your website or web application allows users to login, make sure you have tested this process. In addition we recommend relaunching the test after application updates. Load testing the process is no less important and should be done when you reach a significant number of users. Do it before an expected traffic increase due to the marketing efforts, seasonality, or if spikes of traffic are in the nature of your business. We also highly recommend running login process load tests regularly to always be sure application is ready to handle many users logging in concurrently. With your user count increasing, you should also consider scaling up your load tests accordingly.

With tests created in Loadero, rerunning them and validating your application is ready to serve hundreds or thousands of users logging in is a matter of minutes and just some clicks. Start creating your tests now by signing up for a free trial account and don’t hesitate to contact our helpful support team if you have questions.

A beginner’s guide to test automation with Javascript (Nightwatch.js). Part 1.

Loadero — Wed, 23 Dec 2020 09:06:57 +0000

Introduction to Nightwatch

Welcome to the “A beginners guide to test automation with Javascript(Nightwatch.js)”! This is a three-part blog series to kick start writing web UI automation using Nightwatch.js. In the following parts you will learn the following:

Introduction to Nightwatch.js
How to write a script in Nightwatch.js
How to make your script better in Nightwatch.js

Sounds exciting? Let’s get started!

Writing a script in JavaScript (JS) and Nightwatch.js, aka Nightwatch (NW), is simple once you get started and this guide will help anyone regardless of their skill level. This framework allows writing efficient and easy-to-understand UI automation code with simple configuration. This doesn’t mean that the framework is limited by its rather vast capabilities because it can be easily extended and configured to fit your needs.

This article will be divided into 3 parts – feel free to skip to any part depending on your NW experience:

-What is Nightwatch.js and how to set it up?
-How to write a script?
-How to deploy your test script for cloud-hosted testing?

Code used in this article can be found in Loadero’s public GitHub examples repository here.

Prerequisites

Text editor of your choice (we prefer Visual Studio Code).
Node.js (the latest version is preferable, in this example 'v14.15.0' will be used).
Google Chrome and Firefox browsers.

What is Nightwatch.js and how to set it up?

What is Nightwatch.js?

Nightwatch.js is an open-source (check their GitHub repo and their home page) end-to-end framework that was initially released in 2014 by Andrei Rusu. Since then it has grown dramatically and is still moving forward. We at Loadero are avid Nightwatch.js users and support it in our test scripts from the very first day of our product.

Set up

When starting out, it can be a good thing to run tests locally instead of running them in some cloud service because there is a high chance that a small but critical mistake is made – typo in variable name, incorrect syntax and so on – that will make your tests fail. When tests are run locally, you can easily spot errors in your code, fix them, check once again and, when everything looks great, run the code in some cloud service to gather valuable metrics.

Let’s start by creating a new directory in our home directory where we will host our project locally. To do so, enter the following commands in your terminal:

For Mac/Linux OS:

cd ~
mkdir nightwatch-test
cd nightwatch-test

For Windows:

cd\
mkdir nightwatch-test
cd nightwatch-test

Create package.json file for easier dependency management with default values with the following command

npm init -y

Now let’s download necessary NPM dependencies (Nightwatch, GeckoDriver and ChromeDriver) with the following command which will install the latest available versions of browser drivers and Nightwatch v.1.5.1.

npm install nightwatch@1.5.1 geckodriver chromedriver -D -E

Important: If the non-latest browser versions are used, check ChromeDriver and GeckoDriver supported browser versions.

Tip: We recommend using some kind of code formatter to keep the code consistent. We like Prettier that can be installed both as a code editor extension and NPM global package. Here’s is the .prettierrc configuration file that will be used throughout this
article:

{
 "proseWrap": "always",
 "overrides": [
   {
     "files": "*.js",
     "options": {
       "tabWidth": 4,
       "arrowParens": "avoid",
       "trailingComma": "none",
       "singleQuote": true,
       "printWidth": 100,
       "quoteProps": "consistent"
     }
   }
 ]
}

Now, let’s open the current directory in the code editor of our choice and modify package.json scripts for easier test execution

 "scripts": {
   "test": "nightwatch"
 },

Now, when you enter in the terminal npm test -- --version (yes, 2 --), you should see your Nightwatch.js version and a link to the appropriate changelog. Create a Nightwatch configuration file named nightwatch.conf.js and then paste the following code:

module.exports = {
    src_folders: ['tests'],
    webdriver: {
        start_process: true,
        port: 4444
    },
    test_settings: {
        default: {
            desiredCapabilities: {
                browserName: 'chrome'
            },
            webdriver: { server_path: require('chromedriver').path }
        },
        firefox: {
            desiredCapabilities: {
                browserName: 'firefox'
            },
            webdriver: { server_path: require('geckodriver').path }
        }
    }
};

As the last step tests folder must be created so when we run npm test command all tests in that folder will be run.

The project structure at this point should look like this:

How to write a script?

At once, we can write a Nightwatch.js test. Let’s use DuckDuckGo as an example website for our test script. Let’s create a file duckduckgo_test.js in our tests directory. Now we have to export a CommonJS module object that has one function assigned to the only property – test. This object will be used for Nightwatch.js test execution.

module.exports = {
   test: client => {}
};

Tip: In this example both regular and arrow functions will work but we prefer arrows functions more. If you use regular functions, they can be written like this with shorthand object property:

module.exports = {
   test: function(client) {}
}

To begin with, we have to open DuckDuckGo homepage and wait until #content_homepage (main app container) element loads as a way to determine that the website is ready to be interactable. I usually set wait timeout to 10 seconds (it has to be passed in milliseconds, i.e., 10 * 1000) – if the website doesn’t load within the timeout, the test fails. In Nightwatch.js, these are 3 simple lines.

module.exports = {
    test: client => {
        client
            .url('https://duckduckgo.com/')
            .waitForElementVisible('#content_homepage', 10 * 1000);
    }
};

Now let’s run our first Nightwatch.js test, shall we? To do so we will use the test command defined in package.json. All we have to do is to enter the following line in the terminal and press Enter:

npm test

What just happened? You may have noticed that a Chrome browser window pops up for a split second and then closes and some output in the console that says something like this:

[Duckduckgo Test] Test Suite
============================
ℹ Connected to localhost on port 4444 (1597ms).
  Using: chrome (87.0.4280.88) on Mac OS X platform.
Running:  test
✔ Element <#content_homepage> was visible after 21 milliseconds.
OK. 1 assertions passed. (832ms)

Since we didn’t really see anything, let’s add a 5-second pause after the test, like so:

module.exports = {
    test: client => {
        client
            .url('https://duckduckgo.com/')
            .waitForElementVisible('#content_homepage', 10 * 1000)
            .pause(5 * 1000);
    }
};

Tip: A test might execute so fast that you might not even notice it or it might not have gathered enough data, for example, WebRTC data, which is collected during audio/video call, might not be collected if the user enters the audio/video room only for a split moment. For this reason, it is recommended to have a pause in the script for proper execution. Because pauses are a little bit more complex topic than a tip in the post, they will be covered in this series’ future articles.

Now let’s go through these 4 lines of code to understand what is happening here.

client – the Nightwatch.js object containing functions for executing our test. Check Nigthwatch documentation for their API reference.
.url('https://duckduckgo.com/') – opens provided URL, in this case, DuckDuckGo homepage.
.waitForElementVisible('#content_homepage', 10 * 1000) – waits until #content_homepage element is visible. If the element is not visible within 10 seconds (10 * 1000ms), then the script stops its execution. By the way, this is both an action (wait for some time) and an assertion – check if something is correct, (check if the element is visible), that’s why we see OK. 1 assertions passed. (1.211s) message in the console.
.pause(5 * 1000); – pauses script execution for 5 seconds. In contrast to waitForElementVisible(), this is just an action which doesn’t assert anything. Now when we run our script, we can see DuckDuckGo’s homepage for 5 seconds and the browser window closes. Alright, the webpage is opened and is ready for the next actions, what now? Let’s see how to send input to a search field and search for “Nightwatch.js”. To do so we need to locate the necessary element. We can do so by using the browser “Inspect element” tool and copying CSS selector. But that’s a horrible approach since auto-generated selectors often are way more verbose than needed and that decreases selector and their element traceability, especially, when you have lots of selectors.

Tip: Manually writing a good selector that matches exactly 1 element is the best practice because it increases test stability. Preferably, elements should be located using CSS selectors, and, if that’s not possible, only then using XPath. We suggest checking out these websites to quickly learn how to locate elements: CSS diner and XPath diner.

Once a proper element selector has been identified, let’s update our script to send a value to our selector and after that send ENTER button keypress to trigger the search. For that we will use Nightwatch’s .setValue() function where as the first argument the selector is passed and as the second a string value that will be sent to the selector. You can read more about the function here.

module.exports = {
    test: client => {
        client
            .url('https://duckduckgo.com/')
            .waitForElementVisible('#content_homepage', 10 * 1000)
            .setValue('#search_form_input_homepage', 'Nightwatch.js')
            .setValue('#search_form_input_homepage', client.Keys.ENTER)
            .pause(5 * 1000);
    }
};

In a similar fashion, .setValue() can be used to send all kinds of inputs to different fields, e.g., account credentials in login forms, keypresses to test the support for shortcuts and closing pop-ups with Escape keypress. And as a next step, we can assert that the first search result title contains “Nightwatch.js”. To do so, will use one of many Nightwatch.js assertion functions – .assert.containsText(), you can read more about the function here. To learn more about assertions and verifications, check out our blog post.

module.exports = {
    test: client => {
        client
            .url('https://duckduckgo.com/')
            .waitForElementVisible('#content_homepage', 10 * 1000)
            .setValue('#search_form_input_homepage', 'Nightwatch.js')
            .setValue('#search_form_input_homepage', client.Keys.ENTER)
            .assert.containsText('#r1-0 .result__title', 'Nightwatch.js');
    }
};

After running the updated script, you should see 2 assertions in your console log output. If assertion passes, let’s click on the link first link. In Nightwatch, it is really simple and can be done using one .click() function call where as the only argument the selector has to be passed. To read more about the function, check out Nightwatch documentation about it.

module.exports = {
    test: client => {
        client
            .url('https://duckduckgo.com/')
            .waitForElementVisible('#content_homepage', 10 * 1000)
            .setValue('#search_form_input_homepage', 'Nightwatch.js')
            .setValue('#search_form_input_homepage', client.Keys.ENTER)
            .assert.containsText('#r1-0 .result__title', 'Nightwatch.js')
            .click('#r1-0 .result__title')
            .pause(5 * 1000);
    }
};

As the final step, let’s take a screenshot of our result using .saveScreenshot() function. All we have to pass to the function is the screenshot’s name.

module.exports = {
    test: client => {
        client
            .url('https://duckduckgo.com/')
            .waitForElementVisible('#content_homepage', 10 * 1000)
            .setValue('#search_form_input_homepage', 'Nightwatch.js')
            .setValue('#search_form_input_homepage', client.Keys.ENTER)
            .assert.containsText('#r1-0 .result__title', 'Nightwatch.js')
            .click('#r1-0 .result__title')
            .saveScreenshot('NightwatchJS.png')
            .pause(5 * 1000);
    }
};

After you run your script, you should see the NightwatchJS.png file in nightwatch-test directory.

That’s about it. These are the mere basics of Nightwatch.js to run it locally but now let’s see how to “deploy” your script to Loadero which will give you additional functionality.

Deploy your test script to Loadero

Why deploy your scripts to Loadero in the first place?

You just wrote a simple functional script that checks basic DuckDuckGo functionality. But that’s about it if you run your scripts only locally. By running your script in a cloud platform such as Loadero, you get so many more features than the local environment because you can:

run multi-region test in many countries to test the website’s behavior based on geolocation;
even modify network conditions without any complicated networking setup to test the website’s behavior, for example, in slow 3G network conditions;
effortlessly test the website on multiple browsers and their various versions;
easily convert your functional script into a load test in a few minutes;
run tests with massive number of participants without any system maintenance; Check out our features page to see more information for things you can do in our cloud-platform.

How to deploy Nightwatch.js scripts to Loadero

Before deploying the script, check if you have all the necessary project structure to run your tests (in this case 1 participant with default network configuration will be enough). This blog post might help to get started – Performance Testing Example Using Loadero. If everything is done, click on the “edit” button on your test to update its script. Since Loadero extends vanilla Nightwatch.js functionality with various custom commands and has additional steps inside module.exports to provide you with insightful data that can be analyzed after test execution, you can’t just copy-paste the script from our local text editor. Although, this makes our job even easier – all we have to do is to write one function that will be exported for test execution. So we should update our script like this:

client => {
    client
        .url('https://duckduckgo.com/')
        .waitForElementVisible('#content_homepage', 10 * 1000)
        .setValue('#search_form_input_homepage', 'Nightwatch.js')
        .setValue('#search_form_input_homepage', client.Keys.ENTER)
        .assert.containsText('#r1-0 .result__title', 'Nightwatch.js')
        .click('#r1-0 .result__title')
        .saveScreenshot('NightwatchJS.png')
        .pause(5 * 1000);
}

There is one additional edit we have to do. To save your screenshot during the test, we have to use Loadero’s custom command .takeScreenshot() and provide a screenshot name (not file path) instead using original .saveScreenshot(). This is necessary so that Loadero can find the screenshot after the run and make it available in the run results. More information about this and other custom commands in Loadero can be found in our wiki. After these little changes your script should look like this:

client => {
    client
        .url('https://duckduckgo.com/')
        .waitForElementVisible('#content_homepage', 10 * 1000)
        .setValue('#search_form_input_homepage', 'Nightwatch.js')
        .setValue('#search_form_input_homepage', client.Keys.ENTER)
        .assert.containsText('#r1-0 .result__title', 'Nightwatch.js')
        .click('#r1-0 .result__title')
        .takeScreenshot('NightwatchJS.png')
        .pause(5 * 1000);
}

Now we can copy-paste it to Loadero and update our test configuration accordingly, i.e., set proper participant timeout and start interval. Since our test consists of multiple pauses/timeouts and multiple commands, we should take into consideration maximum possible test execution time. The general formula is

PARTICIPANT_TIMEOUT = TOTAL_PAUSE_TIME + TOTAL_TIMEOUTS_TIME + 1 SECOND * COMMAND_COUNT

In our case “Participant timeout” = 5 + 10 + 1 * 7 = 00:00:23 and since we are having only 1 participant in this test, we can set “Start interval” to 00:00:01. All other test parameters are up to you. In the end, your test should look something like this:

Now click “Save & Exit”, click “Run test” and you just run your very first Nightwatch.js test! Go to all results view by clicking “Results” button on the left side bard. Test execution should pass, but in case it fails, reread this article, check your script again and if this doesn’t help you, check our recommendations on debugging Selenium test scripts.

Conclusion

Nightwatch.js is an amazing framework which allows you to write web UI automation quickly and simply. In case you are really into configuring every little detail for your test runner, it has no compromises and enables you to do so as well. Once you have written all the code and validated it locally for any potential errors, you can deploy the script to Loadero and utilize all the stunning features it has to offer – starting from the success rate for your test and ending with detailed metrics, graphs and logs for each of your configured test participants.

6 Ways of Selenium Test Scripts Debugging

Loadero — Thu, 26 Nov 2020 09:15:12 +0000

Not always writing test scripts goes as smoothly as planned. Sometimes even seemingly easy tests take way too long to make them right. Especially when just starting to write tests there are a ton of potential issues that can pop up. These issues can be in the website itself and also in the written test script. Debugging is one of the most important skills any automation tester should learn. There are many ways of test script debugging. In this blog we will show you some so you can debug your automated testing scripts right now.

Local testing

One of the easiest ways to validate your script is by running it locally on your own machine. In case the automated tests are running on remote devices or on a cloud platform like Loadero, extra visual validation and manual interference could give a hint where the issue might lay. It is quite important that the script runs in the same environment as the remote device, so there are no errors due to inconsistencies in the configuration. Make sure to pay attention to this when you start your test script debugging with local testing.

In case you want to try this method, here are the links where to get both of the frameworks that we support in Loadero:

Verbose Selenium logging

If verbose logging is enabled, then more action by the framework will be logged. These messages sometimes contain API calls and their responses (such as for Nightwatch). From their responses it’s possible to determine which element was manipulated or found. For example, such logs could indicate that a browser alert was triggered or an element is not available. In the other side TestUI with enabled verbose logging will log every action done in the test, but without enabling it – only test status will be logged at the test end.

The only downside for enabling verbose logging is that it will clutter up the logs and make them unintelligible.

This can help with test script debugging a lot, and in Loadero all performance test participants have access to verbose Selenium logs in the performance test mode. Check out our Wiki page to get more information about test modes!

Console logs

In case of test failure browser console logs have to be always checked. Most of the time they are empty or contain just a couple of warnings. But if the website has thrown any errors during test execution – console logs is where you will find them. Those errors are on the website side and can be caused by a million reasons. But during automated tests they usually occur on button clicks, because that is an action that triggers extra functionality on the website’s side.

Session recording

This is a way of test script debugging that we are proud to offer to our service users. Loadero has a test mode that will video record the whole test duration. So it’s possible to visually validate the test actions and make sure no unexpected elements or alerts appear. By using session recording it’s easier to detect problems with changing UI, for example unexpected redirects. Such issues potentially can be easily missed in logs. In case of a page redirect there could be no logs about it and the engineer is left in the dark about what caused the test to fail.

In addition session recordings can be saved and used later for investigation of what caused the issues and search for visual improvements, for example in group call video quality. This recording could give an indication of UI usability. Recording a session with network conditions set can also give insights on application behavior, when a user has worser connection. Learn more about testing with different network conditions from this blog post. User experience is very important, after all, the visual functionality is the first thing that the user encounters.

Recording a video has impact on the machine resources, so when using session recording keep in mind that the system could possibly be slower to compensate for the extra load, especially if the website that is being tested itself is very resource intensive.

Screenshots

Without a doubt, the easiest and quickest way to debug your automated testing script is using screenshots. Both TestUI and Nightwatch support taking screenshots during script execution. When using Loadero these screenshots can be taken using our custom commands. If you already have some tests in Loadero or just planning to create some, make sure to add the commands to take screenshots. Our Wiki explains how to achieve that in NightwatchJS. After test execution the screenshots can be found in participant results view under "Artifacts" tab. More on screenshots and Loadero test results view is explained in this blog post about results reports.

In the real world, there is no downside for using screenshots. They don’t require a lot of machine resources and do not interfere with the test itself. Matter of fact, we would recommend at least creating screenshots at the main possible problematic points. That can possibly save time and costs for relaunching the test.

Check the scenario manually

While writing tests we strongly recommend opening the website using a fresh incognito or private tab. This helps to avoid all possibilities for previously set settings and cache. The simplest things like cookie banners or pop-ups are often forgotten. But they are usually shown for new visitors and such elements could be in the way of the website’s UI. Also things like captcha nowadays tend to appear more for new visitors. If this is the case, sadly there is not out of the box way to bypass it.

There are many more techniques for your test script debugging, we explained just six to help you start. Some of the described approaches can be used not only for debugging purposes. Session recordings and screenshots are also very useful in performance testing as well. Sign up for our free trial and run multiple performance tests free of charge. Use the debugging techniques to prepare your tests for future large scale testing. If you will require any assistance during exploration, make sure to contact our helpful support team.

DEV Community: Loadero

Migration Of An Application Frontend To TypeScript

Write guidelines and plan migration

Setup the project

Migration

Caveats

How To Test Screen Sharing Feature In Web Applications

How screen sharing works in web applications

Why should one test screen sharing feature

How screen sharing impacts machine resource usage

How to test screen sharing

Verifying that screen sharing works in your test

Local Tests With Selenium And Python Browser Automation

Python environment

Automation environment

Running a test locally

Using Python browser automation test script

Test Automation With Python

Test Structure

Driver

Navigation

Clicks

Input Text

Retrieving Content of an element

Putting Everything Learned To Use

Testing A Communications Platform’s (CPaaS) Performance

What’s important in testing a communications platform

Types of tests to run

Steps to start testing a communications platform

CSS Selectors And XPath Locators In Selenium Test Automation Scripts

CSS selectors

Simple CSS selector

Pseudo selectors

Attribute CSS selectors

XPath

Conclusion

A Beginner’s Guide To Test Automation With Javascript (Nightwatch.js). Part 4.

How to make your script better in Nightwatch.js

Prerequisites

Consistent code style

Arrow functions

Destructuring

Element selector usage

CSS selectors

XPath selectors

Web element ID

Code organization

Subfunctions

Selector objects

Extracting hardcoded values into variables

Summary

A Beginner’s Guide To Test Automation With Javascript(Nightwatch.js). Part 3.

Callback functions and command queue in Nightwatch.js

Callbacks

Nightwatch.js callbacks

Command queue

Variable resolution in the queue

Summary

A Beginner’s Guide To Test Automation With Javascript (Nightwatch.js). Part 2.

How to write a script in Nightwatch.js

Prerequisites

The most useful Nightwatch.js commands

.url()

.waitForElement…()

.click()

.setValue()

.pause()

.saveScreenshot() / .takeScreenshot()

.perform()

The final script##

Summary

How To Create An Automated Login Page Test And Use Multiple Credentials

Login test data preparation

Using hardcoded credentials

Using externally hosted credentials

Loadero test and participant setup

Conclusions

A beginner’s guide to test automation with Javascript (Nightwatch.js). Part 1.

Introduction to Nightwatch

Prerequisites

`.url()`

`.waitForElement…()`

`.click()`

`.setValue()`

`.pause()`

`.saveScreenshot()` / `.takeScreenshot()`

`.perform()`