pytest Report Generation For Selenium Automation Scripts

Test automation frameworks such as pytest are very powerful in automating tests when used along with the Selenium, the test results generated by these tests can become more insightful when you combine testing with reporting feature supported by the test framework.

In case you are new to pytest framework checkout this video to know more about pytest framework & its installation.

The primary advantage of the report feature is that the output gets generated in a simpler & readable format, mostly in the form of HTML or XML file. Since these file formats are widely used, it makes the task of decoding the output easier. In this article, we have a look at how you can use pytest report generation with Selenium. We will also look at ways to enhance the reporting functionality by making use of LambdaTest APIs for Selenium.

Importance Of Generating Selenium Reports

Before taking a deep dive into pytest report generation, it is important to understand the relevance of generating Selenium reports in the context of a web development project/product.

Test your native, hybrid, and web apps across all legacy and latest mobile operating systems on the most powerful Android emulator online.

Tracking & Readability

When a tester performs the test execution, the test results along with necessary important information (like test cases passed, release on which test was performed, details about input arguments, detailed information about failure scenarios, etc.) might be required to be maintained for future reference. This can be preserved in a popular format (HTML, XML, etc.) in the form of Selenium reports making it easy to track the code maturity over a period of time.

Ease Of Integration & Execution

Whether it is pytest report generation or Selenium reports for any other kind of test automation framework, integrating a module that supports the generation of Selenium reports e.g. pytest-html for pytest report generation is a simple task. All you have to do is install the module and import the same in your test code. If you are using command line parameters to execute the test code, generating reports is just an addition of a couple of parameters e.g. –html=pytest_selenium_test_report.html in pytest.

Visual Content For Easy Comparison

There might be scenarios where you would like to store the results of the UI tests and Selenium reports are the easiest way in which you can save this information.

Integration With CI/CD tools

Majority of the Selenium reports module e.g. pytest-html used for pytest report generation can be seamlessly integrated into your CI/CD (Continuous Integration/Continuous Delivery) process tools such as Jenkins, CircleCI. Hence, you do not any specialized changes in your build & integration process.

Back To pytest Report Generation Using pytest-html

Pytest report generation for your Selenium scripts can work like magic in terms of saving you the time & an effort by maintaining data in an organized way, so it’s easy to visualize & comprehend where and when your scripts are failing.

pytest-html – Installation & Usage

Though there are different mechanisms to generate Selenium reports in pytest, pytest-html is the most preferred option for pytest report generation. The pytest-html module was initially a sub-part of the pytest-mozwebqa module which later on was made a separate plugin/module which had the sole purpose of generating Selenium reports (an alternative to JUnit report or console output).

If you are new to pytest, then I would recommend you have a look at one of my previous articles to help you run your first automation script using pytest with Selenium WebDriver.

With that said, let’s get started with pytest report generation using pytest-html. Execute the below command on terminal for installing the pytest-html module.

pip install pytest-html

Below is a snapshot of the command in execution.

For implementation purposes, we are making use of the Community version of the PyCharm IDE which can be downloaded from here. In order to demonstrate the usage of pytest-html for generating Selenium reports, we have a look at a pytest example which makes of pytest fixtures, & Selenium Webdriver.

# Import the 'modules' that are required for the execution

import pytest
import pytest_html
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from time import sleep

#Fixture for Firefox
@pytest.fixture(params=["chrome", "firefox"],scope="class")
def driver_init(request):
    if request.param == "chrome":
        web_driver = webdriver.Chrome()
    if request.param == "firefox":
        web_driver = webdriver.Firefox()
    request.cls.driver = web_driver
    yield
    web_driver.close()

@pytest.mark.usefixtures("driver_init")
class BasicTest:
    pass
class Test_URL(BasicTest):
        def test_open_url(self):
            self.driver.get("https://www.lambdatest.com/")
            print(self.driver.title)

            sleep(5) 

As seen in the above example, the URL under test https://www.lambdatest.com is opened in the Chrome & Firefox browser. You can also observe, the pytest-html module is imported to enable the reporting functionality. You can make use of the following command to execute the same.

py.test.exe --capture=no --verbose --html= test_selenium_webdriver-1.py test_pytest-html.py 

The results of execution are below.

Below is the HTML report that contains information about the tests & their results.

pytest-html – Reports Enhancement

The appearance of the report can be changed by passing Cascaded Style Sheets (CSS) options using the –css option. Since multiple CSS settings can be applied, they would be applied in the order in which the options are received.

pytest --html=pytest_selenium_test_report.html --css= 

If you plan to add more details to the HTML report, you can do the same by creating extra list on the report object. Listed below are some of the options that can be used for adding more informative content in the pytest Selenium report.

While inputting the image path, you can make use of a relative path as well as an absolute path. You can also modify the results table to improve the readability or to add more information. You may refer

https://pypi.org/project/pytest-html/ for a detailed procedure on modifying the results table.

This PyTest Tutorial for beginners and professionals will help you learn how to use PyTest framework with Selenium and Python for performing Selenium automation testing.

Online emulators from LambdaTest allows you to seamlessly test your mobile applications, websites,and web apps on mobile browsers and mobile devices.

Detailed Selenium Reports For Effective Cross Browser Testing Analysis

One shortcoming that every web development project/product witness during the cross browser testing phase is the extent of testing that can be performed on different types & versions of browsers(Firefox, Internet Explorer, Microsoft Edge, Chrome, etc.). Things become more complicated when you add the extra combination of operating systems(Windows, Linux, macOS, iOS, Android, etc.) and various devices i.e. mobile, desktop, tablets. Rather than attempting to scale your existing infrastructure to fulfill the demands of the test team, it is recommended to perform cross browser testing on the cloud. The primary advantage of such an approach is that it is more scalable & economical than setting up local infrastructure. Also, you can speed up the automated cross browser testing process can be improved by making use of Parallel testing in Selenium.

LambdaTest provides an easy to scalecross browser testing platform on the cloud that offers more than 2,000+ real browsers & operating systems online to help you expand your cross browser testing coverage to full. You can also execute automated Selenium tests using our robust, easy to scale, & faster Selenium Grid on cloud.

RUN YOUR SELENIUM SCRIPTS ON CLOUD GRID

2000+ Browsers AND OS

FREE SIGNUP

During the product development & testing phase, you may want to verify the features on different browsers and having the test execution results on the cloud makes tracking easier. Hence you can easily keep a track of the test analytics & test results and share with your team members as well.

LambdaTest has provided Selenium API that addresses all these concerns and using that API, you can manage your test builds, test sessions, track test analytics, fetch logs, fetch the screenshots for every test, etc directly from their platform to your preferred storage facility, even without logging into your LambdaTest account. It provides a file in the JSON format which contains syntax for every browser and the corresponding browser version along with the operating system on which the tests are performed.

The API URL that can be used for getting more out of test reporting is below

https://api.lambdatest.com/automation/api/v1/

In order to use the API, all you need to do is append the corresponding end-point to the end of the Computed URL and issue a GET/POST. For example, you can make use of “/builds” to get details of every build performed in that particular account. The computed URL would be:

https://api.lambdatest.com/automation/api/v1/builds

To get information about the number of test sessions. Computed URL:

https://api.lambdatest.com/automation/api/v1/sessions

You can even check the information at a session level using the session-id to get information related to a particular session. Computed URL:

https://api.lambdatest.com/automation/api/v1/sessions/{session-id}

Refer to our blog on LambdaTest Selenium API for more details related to all the end-points that can be used along with the LambdaTest API.

LambdaTest API helps you to extract, and analyze your Selenium test reports with as granular detail as you want.

Detailed pytest Report Generation From LambdaTest

Now that you have knowledge about the supported endpoints, let’s have a look at how these can be used in the right way for pytest report generation using LambdaTest. To get started, you have to login to API reference document with your LambdaTest username and access key. You can find these details from LambdaTest Automation Dashboard by clicking the key icon.

Once you have your user-name & access key handy, you need to authorize the Lambdatest API using those credentials. Once you authorize, GET, POST, etc. requests can be triggered using the API. You can access details about the Builds, Sessions, and Tunnels that are used during the automated cross browser testing process using the Selenium Grid.

For example, there are a couple of automated tests that we had performed, details of which can be viewed by clicking the Automation tab on the left-navigation panel on LambdaTest and visiting the Timeline.

For demonstration, we select a particular build e.g. BuildID = 11254 for using the automation API. The Computed URL is of the below format:

https://automation.lambdatest.com/logs/?testID={test-id}&build={build-id}

We initially post the existing source code for executing it on the Selenium Grid on LambdaTest. Important Note: You will have to replace the user_name and app_key with the username & access key which as stated above, are present in the Automation Dashboard.

# Import the 'modules' that are required for execution

import pytest
import pytest_html
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
from time import sleep
import urllib3
import warnings

#Set capabilities for testing on Chrome
ch_caps = {
    "build" : "Testing using Chrome on Windows",
    "name" : "Testing on Chrome using Selenium Grid",
    "platform" : "Windows 10",
    "browserName" : "Chrome",
    "version" : "71.0",
    "selenium_version" : "3.13.0",
    "chrome.driver" : 2.42
}

#Set capabilities for testing on Firefox
ff_caps = {
    "build" : "Testing using Firefox on Windows",
    "name" : "Testing on Firefox using Selenium Grid",
    "platform" : "Windows 10",
    "browserName" : "Firefox",
    "version" : "64.0",
}


# Details can be sourced from https://automation.lambdatest.com/
user_name = "user-name"
app_key = "access key"

#Fixture for Firefox
@pytest.fixture(params=["chrome", "firefox"],scope="class")
def driver_init(request):
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
    remote_url = "https://" + user_name + ":" + app_key + "@hub.lambdatest.com/wd/hub"
    if request.param == "chrome":
    #Remote WebDriver implementation
        #web_driver = webdriver.Remote(
            #command_executor='http://107.108.86.20:4444/wd/hub',
            #desired_capabilities={'browserName': 'chrome', 'javascriptEnabled': True})
        web_driver = webdriver.Remote(command_executor=remote_url, desired_capabilities=ch_caps)
    if request.param == "firefox":
    #Remote WebDriver implementation
        #web_driver = webdriver.Remote(
        #    command_executor='http://107.108.86.20:4444/wd/hub',
        #    desired_capabilities={'browserName': 'firefox'})
        web_driver = webdriver.Remote(command_executor=remote_url, desired_capabilities=ff_caps)
    request.cls.driver = web_driver
    yield
    # web_driver.close()
    #print(self.driver.title)
    sleep(5)
    web_driver.close()
    web_driver.quit()

@pytest.mark.usefixtures("driver_init")
class BasicTest:
    pass
class Test_URL(BasicTest):
        def test_open_url(self):
            self.driver.get("https://www.lambdatest.com/")
            print(self.driver.title)

            sleep(5) 

Every test session has a unique Session_ID associated with it. Another important parameter that needs to be kept in mind is the Test_ID. A single Test_ID can have multiple Session_ID(s) associated with it. You need to go to the COMMAND section of a particular test to get the Session_ID of the test, a sample is shown below:

In order to demonstrate the LambdaTest APIs, we extract the information about the first 10 builds with the status as Completed, Timeout, and Error. For this purpose, we make use of “/builds” API. We perform the following steps to achieve our target

1. Login to LambdaTest API reference document using the Username & Access Key obtained from your Automation Dashboard.

1. Once logged in, you have to head to the Build section which has the option to get build related information /builds Fetch all builds of an account and press on the button named Try It Out. Next, enter the relevant details in the text box titled Limit i.e. 10 since we need the information about the first 10 builds and enter completed, timeout, error in the text box titled status.

1. Press the Execute Button and you should get the results in the JSON format. You should also have a look at the Responses section which shows the CURL API which you can use in your code. The response code can be 200 (successful operation), 400 (Invalid session id value) or 401 (Access denied. Auth error.) Below is a screenshot of the response that we received for the query that we performed. The server response is 200 which means that the execution was successful.

1. You can use the website https://curl.trillworks.com/ to convert the CURL response into equivalent Python code. You are open to using any other website that offers free conversion or any tool that offer you the same.

1. Now that we have the bare skeletal of the Python code ready, we make the relevant changes to extract the required information from the JSON output i.e. Print Build_ID and Corresponding Status in the terminal. Along with fetching the matching build information, we also change the Build Name for the Build_ID 11254. For doing the same, we make use of https://api.lambdatest.com/automation/api/v1/builds/{Build_ID} API were Build_ID = 11254. You need to repeat the same steps (1-4) for this requirement as well so that you get the equivalent Python code for performing this operation.

Below is the final code which shows the LambdaTest API in operation

# Requirements

# Fetch first 10 builds which have status as completed, error, timeout
# Once the details are out, change the Build Title of the build_id 11254

# Refer https://www.lambdatest.com/support/docs/api-doc/#/Build/builds for more information
import requests
import json

# Equivalent Python code from https://curl.trillworks.com/
headers = {
    'accept': 'application/json',
    'Authorization': 'Basic aGltYW5zaHUuc2hldGhOVI4bHdCc1hXVFNhSU9lYlhuNHg5',
}

params = (
    ('limit', '10'),
    ('status', 'completed,timeout,error'),
)


# Updated build information for build 11254
headers_updated_build = {
    'accept': 'application/json',
    'Authorization': 'Basic aGltYW5zaHUuc2hldGhOVI4bHdCc1hXVFNhSU9lYlhuNHg5',
    'Content-Type': 'application/json',
}

data_updated_build = '{"name":"Updated build details from prompt"}'

response = requests.get('https://api.lambdatest.com/automation/api/v1/builds', headers=headers, params=params)

print(response)

json_arr = response.json()

# Print the build_id matching our requirements and Change build title of build_id 11254

for loop_var in range(10):
    build_id = ((json_arr['data'][loop_var])['build_id'])
    test_status = ((json_arr['data'][loop_var])['status_ind'])

    if build_id == 11254:
        response_updated_build = requests.patch('https://api.lambdatest.com/automation/api/v1/builds/11254', headers=headers_updated_build, data=data_updated_build)
        print(response_updated_build)

    print ((build_id), (test_status)) 

Below is a screenshot of the execution. As seen below, we have received Response as 200 two requests:

https://api.lambdatest.com/automation/api/v1/builds

https://api.lambdatest.com/automation/api/v1/builds/11254

This indicates that the operation was successful.

In order to verify whether the Update Build Name operation for Build_ID 11254 was successful, we jump into Automation Logs of LambdaTest and checked the status i.e.

https://automation.lambdatest.com/logs/?testID=CKSVC-J7YFC-&build=11254

Let’s take a look which are the most wanted automation testing tools in 2023 that have climbed the top of the ladder so far?

Conclusion

Selenium reports, if used effectively can be of great help to keep a check on the overall test activities and their results. For pytest report generation, we can make use of pytest-html module for cross browser testing with Selenium Grid. Information presented in the Selenium reports can be enhanced by making use of LambdaTest APIs which provides you minute details about build/session/tunnels for your Selenium automation scripts performed on LambdaTest server.

Based on your requirements, you can export this information in the HTML report and simplify your entire browser compatibility testing process. Using LambdaTest APIs, you can reduce the time involved in automation testing, cross browser testing, and maintenance of test reports that carry relevant information about software builds/releases.

Originally Published: LambdaTest