Use Python to run parallel (and parametrized) Robot Framework tests

#python #testing #concurrency #robotframework

Disclaimer: I am neither an expert regarding the Robot Framework nor parallelization with Python. I simply found this possibility very use- (and learn-)ful and would like to share it whilst trying to explain how it works.
If you are looking for more sophistiacted ways to execute concurrent Robot tests you should take a look at the pabot library or Taurus.

Requirements:

Knowledge of (and/or interest in) the Robot Framework
Basic Python knowledge

What?

... how to use concurrent subprocesses in Python
... to run our Robot test multiple times in parallel
... with different parameters without writing multiple tests (for example to allow testing with different test users)

Why?

   👾 simple way performing stress tests against a system
   🐢 speed up your testing
   🐿️ fun

Excursion: Robot Framework

From robotframework.org:

Robot Framework is a generic open source automation framework for acceptance testing, acceptance test driven development (ATDD), and robotic process automation (RPA).

Its Keyword based syntax is easy to learn and makes it possible to write human readable tests very fast.

I like to use it together with the Selenium Library which provides a rich set of keywords for testing web pages and applications.
In cases where those are not sufficient Robot Framework allows you to extend its functionality by:

Using additional libraries
Implementing your own keywords in Java or Python
Executing Javascript via Selenium Keywords 🎉
...

See the official documentation for more information and posibilites.

Let's start ⌨️

Parameterizing Robot tests

We start with a single Robot test file called suite.robot containing our test(s). This file contains a global variable for the login name of our test user:

${USER}           test-user
...

Login Test
    Go To    ${HOMEPAGE}
    Wait Until Element Is Visible    username
    Input Text    username    ${USER}
    Input Password    password    ${PASSWORD}
    ...

Via command line options you are able to override the default value for this variable when running the test.

For example

robot --variable USER:another-user suite.robot

runs the test for the user another-user (instead of test-user).

Subprocesses with Python

To make use of the above described options via Python we can use subprocess.Popen (doc). Above example would be translated into:

import subprocess
args = ['robot', '--variable', 'USER:another@user.org', 'suite.robot']
subprocess.Popen(args)

If you try this, you will see that the output of our Robot test gets logged to the console. This is not very helpful if you want to run multiple tests at the same time.
We can disable this behavior by redirecting the output to /dev/null like this:

import os
FNULL = open(os.devnull, 'w')
subprocess.Popen(args, stdout=FNULL)

Another argument for Popen we will use is cwd. It allows us to specify the working directory for the created process.
This is useful since robot will create the log files of our test in the current working directory. If we would run all our tests within the same directory those log files would collide.

In our final script we make sure to create a separate folder per user, containing only the logs of the test for this user:

os.mkdir(userPath)
subprocess.Popen(cmdParts, cwd=userPath, stdout=FNULL)

Put it together

Now we are able to put together a script expecting following files as input:

suite.robot containing our Robot test(s)
user_list.txt containing list of usernames (one per line)

and doing the following:

Create a base folder containing our user folders
Load user_list.txt
For each user:
- a subfolder is created
- a subprocess is created running our test with the users parameter(s) within its subfolder
Wait until all subprocesses finished

Full script:

Possible improvements:

Make it more dynamic: Create input file containing arbitrary variable parameters per test (not only the username)
Live view of each test in the console by redirecting stdout more sophisticated
Thread pool management (for example to restrict maximum number of concurrent subprocesses)
Aggregated report

Thank you for reading :)