DEV Community: RakeshVardhan1

How To Build CI/CD Pipeline With TeamCity For Selenium Test Automation

RakeshVardhan1 — Thu, 24 Nov 2022 08:05:36 +0000

Continuous Integration/Continuous Deployment (CI/CD) has become an essential part of modern software development cycles. As a part of continuous integration, the developer should ensure that the Integration should not break the existing code because this could lead to a negative impact on the overall quality of the project. In order to show how the integration process works, we’ll take an example of a well-known continuous integration tool, TeamCity. In this article, we will learn TeamCity concepts and integrate our test suites with TeamCity for test automation by leveraging LambdaTest cloud-based Selenium grid.

There are numerous best CI/CD tools available for building high-quality code and narrowing the gap between development and impacted teams. Besides establishing a DevOps culture in the organizations, teams enhance it by implementing best CI/CD practices throughout the Software Development Life Cycle(SDLC). These practices help the teams accelerate product development, automate the processes, and improve overall productivity.

If you’re new to Selenium and wondering what it is then we recommend checking out our guide — What is Selenium?

Which are the most wanted tools for automation testing that have climbed the top of the ladder so far? Let’s take a look.

What is Jetbrains TeamCity?

JetBrains TeamCity is a user-friendly and general-purpose CI/CD solution for developers and build engineers. TeamCity helps the team carry out all kinds of workflows and development practices within the SDLC life cycle.

TeamCity comes in two different flavours.

TeamCity Professional

A free version of the software even for commercial use. We can connect up to three build agents and create up to a hundred build configurations.
It is multi-platform, multi-language, and multi-cloud — helps build, test, and deploy apps, packages, and containers of all sorts.

TeamCity Enterprise

An enterprise version of the same software where we can create unlimited build configurations.
Manage hundreds of build agents with a single server. Connect multiple servers into a cluster if required.
Priority customer support from Jetbrains.

What Can You Do With TeamCity?

Let’s look at the following services offered by Teamcity for test automation.

Self-hosted Continuous Integration.
Simultaneously run parallel builds on different platforms and environment configurations.
Configure the TeamCity continuous delivery pipelines via UI or configuration-as-code option using Kotlin DSL.
Customize the statistics for different aspects — build duration, success rate, quality of the code, and custom metrics.
Run the code coverage and duplicates finder for Java and .NET environments.

Note: To know all the features of TeamCity, please have a look at the features overview section of the Jetbrains official site.

TeamCity Architecture

Getting started with TeamCity has never been easier. Unlike some other build servers, TeamCity has distributed grid architecture, i.e., the TeamCity build system comprises two primary components — the server and a farm of build agents.

The TeamCity Server provides the user interface for managing build configurations and managing results. The server doesn’t run any jobs or tests.
The TeamCity Build agent executes build configurations or build jobs and can be installed on-premise or in cloud providers like AWS/Azure/GCP.

TeamCity server initiates a build process whenever a specific condition is met. These conditions might be that there are changes in the VCS that TeamCity has detected. Many other conditions can trigger a build. Once a specific condition is met, the TeamCity server tries to identify the available agents based on the appropriate environment and compatibility. If there are no agents available, the build is added to the build queue. The build will be in the queue waiting for assigning to an agent. Once the build has been assigned to an agent, the agent has to get the sources to run the build.
TeamCity provides two possible options on how the build agent can get sources needed for the build:

Server-side Checkout: TeamCity server exports the required sources and passes them to the build agent.
Agent-side Checkout: The build agent itself checks out the sources before the build.

Once the build agent has all the required sources and files, it starts executing the build steps one after the other sequentially. Each step is represented by a specific build runner in this process — for example, maven, gradle, or sonarqube server, etc.
As the build steps are being executed, the build agent sends all the log messages, code coverage results, testing reports, etc., to the TeamCity server instantly. Because of this, we can monitor the build process in real-time on the server console.
Once all the build steps are executed, and the build is completed, the build agent sends the generated artifacts to the server. These are the files produced by the build — for example, WAR or JAR files, test reports, log data, etc.

TeamCity supports development across all types of platforms, including Windows and Linux, macOS. TeamCity can be easily extended and integrated with other tools that make up the SDLC toolchain.

Some of the integrations and toolchain that TeamCity support is shown below:

TOOL CHAIN	INTEGRATION
Build Runners	Apache ANT, Maven, Gradle, Microsoft .NET, Helm
Testing Frameworks	Junit, Nunit, TestNG, Apache ANT, Maven, MSTest, VSTest, MSpec
Version Control Systems	Github, Gitlab, Bitbucket, Azure DevOps, Git, Subversion, Mercurial, Perforce
Integrated Development Environments	IntelliJ IDEA, Visual Studio, Eclipse
Code Quality	Inspection, Duplicates, Finder, Emma, JaCoCo, FxCop, dotCover, NCover
Cloud	AWS, Google Cloud, Microsoft Azure, VMWare vSphere, Docker, Kubernetes
Defect Tracking Systems	You Track, Jira, Github, Bitbucket, Azure DevOps
Others	Unity, Octopus Deploy, SonarQube, Slack, JFrog, Snyk

Advantages Of TeamCity

TeamCity has the below advantages over other CI/CD solutions.

For Developers	For DevOps Engineers	For Managers	For Admins
Perform Better	Configuration as code	Get a bird’s view of the CI	Zero maintenance
Get started easily	Pipeline Optimization	Organize team’s work	Scalability
Test Intelligence	Templates	Can grow to any scale	Cloud integrations
Real-time reporting	RESTFul API	Benefit from a fast feedback loop	Access Management
Remote run and pre-tested commit	Multiplatform and Multilanguage	Professional technical support	Rich plugin ecosystem

TeamCity Terminologies

To perform test automation with TeamCity, let’s look at the concepts/terminology used within TeamCity.

Terminology	Description
TeamCity Server	The major component in the TeamCity ecosystem that stores all the objects’ settings, manages the build queue, monitors the state of running builds, and performs many other tasks. We can have this server on-premise or in any cloud(AWS/GCP/Azure etc.) or TeamCity Cloud.
Build Agent	Build agent is the software that executes a build process as per the instructions from the server. Typically, it is installed and configured on a separate machine than the TeamCity server. TeamCity build agents can have multiple operating systems, different platforms, and also pre-configured environments.
Project	TeamCity project is a collection of all different build configurations. We can create a project in TeamCity corresponding to a software project, specific release/version of a project, or any other logical grouping as per the product. These projects can be nested and organized into a tree-like structure.
Build Configuration	A build configuration in TeamCity is a collection of settings used to start a build or group the sequence of builds in the UI console.
VCS Root	Whenever TeamCity needs to get the source code, it needs to connect to the version control system(VCS), referred to as a VCS root. TeamCity monitors the changes in VCS and gets the sources for a specific build configuration.
Build Step	Step is the task that needs to be executed, such as compiling the project, running code coverage, invoking the test tools, etc.
Build Trigger	Triggers are used to add builds to the queue. We can add a trigger either when an event occurs (for ex. VCS check-in) or periodically with some configurable interval
Change	Any modification to the source code of the project repository.
Build	It is the actual process of creating an application version. After the build process is triggered, it is added to the build queue and is started when there are agents available to run it. After the build is completed, the build agent sends build artifacts to the server.
Build Queue	Represents the list of builds that were triggered and are waiting to be started by the agents. TeamCity distributes the builds to the available agents as soon as they become idle.
Build Artifacts	After a build is completed, typically, we get files such as WAR/JAR files, installers, log files, etc. Once they are available for download, they are called build artifacts.

Want to know which are the 30 Top Automation Testing Tools in 2022? Let’s take a look.

Mapping Some Concepts Of TeamCity With Jenkins

As we all know, Jenkins is one of the most used CI/CD servers across different teams. Jenkins and TeamCity mainly provide the same set of concepts and functionalities but with slightly different naming conventions.

The following table provides a mapping for some of the Jenkins concepts and terminology to the TeamCity counterparts.

Terminology	Description
Jenkins Master/Node	TeamCity Server
Dumb Slave/Permanent Agent	Agent Pool
Executor	TeamCity Agent
View or Folder	Project
Job/Item/Project	Build Configuration
Build	Build Steps
Pre Step	Build Steps (Partially)
Post-Build Action	Build Steps (Partially)
Build Triggers	Build Triggers
Source Control Management (SCM)	Version Control System (VCS)
Workspace	Build Checkout Directory
Pipeline	Build Chain (via snapshot dependencies)
Label	Agent Requirements

TeamCity Installation

It’s a lot of theory! Now let’s install TeamCity, do some hands-on, and understand each of the concepts in detail.

The TeamCity installation package is identical for both Professional and Enterprise editions. TeamCity can be installed and used in the following ways.

Direct installation package. (.exe for Windows, .tar.gz for Linux/macOS)
Docker Image
TeamCity Cloud — as a managed service

Let’s start with each of the options in detail.

1. Direct Installation Package

On Windows — Navigate to TeamCity official download page and download the installation package. After downloading, you can start using TeamCity under the free Professional license right away. It is enabled by default.

Once the file is downloaded, run the file to start the installation process.
Read and accept the Licence Agreement.
Choose the folder in which you want to install the TeamCity or accept the default path provided. Click on Next.
As discussed above, TeamCity has two components — Server and Agent. Select the components as required to be used on the machine and click on Next.
Since we will use the same machine as the Server and Agent, we will select both options. This approach is only applicable for learning and exploring TeamCity. Click on Next.
Let us choose port 8088 and proceed with the installation.
On the next screen, TeamCity displays the build agent properties with the selected options. As chosen earlier, the server should be running on port 8088 and should be available on URL — http://localhost:8088, and the agent listens on port 9090 for all the server communication. Let’s not change any of these details, as we go with default values and click on Save.
Once clicked on OK, we need to choose a user account to run the server and agent services. We select the SYSTEM account and click on Next.
Select the services to start on the machine.
Click Finish on the installation wizard. TeamCity web interface will be launched in a browser at the URL — http://localhost:8088/.

TeamCity might take some time to load all the components and show the UI. Choose the default data directory or use a custom folder to store all the build configuration and artifacts.

TeamCity stores all the build history and user-related data in a relational database. In the Database Connection setup screen, we can choose the appropriate SQL database. For the demo/training purposes, we could choose the internal database (HSQL) by TeamCity.

Note: It is recommended to configure a database for production purposes before this and configure it with TeamCity as needed.

On Proceed, TeamCity will create a SQL database and provision all the required resources.

Once everything is ready, the License Agreement screen is displayed. Carefully read and accept the agreements, proceed to Next.

TeamCity advised us to create an administrator account. Choose the details appropriately and click on Create Account.

That’s it! The TeamCity installation and initial configuration are complete. We will be redirected to the TeamCity server home page as below.

We could also choose to use the new experimental UI or continue using the classic UI.

We have chosen to use the new UI; hence the below home page will be displayed.

The installation process is pretty similar in Linux and macOS. Please refer to these instructions for the same.

Perform browser automation testing on the most powerful cloud infrastructure. Leverage LambdaTest Automation Testing platform for faster, reliable and scalable experience on cloud.

2. Installation using Docker

TeamCity can also be installed using docker. Login to the machine where Docker is present and follow the below instructions to install TeamCity. If you are not familiar with docker, please refer to these guidelines to install and set up.

Typically, TeamCity is used with relational databases configured separately in production configurations. We will use Postgresql in these installation steps. This is how we are going to configure TeamCity here.

Linux machine1 — used as a server + postgres DB
Linux machine2 — used as an agent

1. Using Linux Machine 1 As A Server + Postgres DB

Pull the postgres image as the root user on the linux machine.

$ sudo su -
$ docker pull postgres
Create a directory named postgres in the docker volumes.

$ mkdir -p $HOME/docker/volumes/postgres
Run the postgres container with the below options.

$ docker run --name pg-docker -e POSTGRES_PASSWORD=start123 -d -p 5432:5432 -v $HOME/docker/volumes/postgres:/var/lib/postgresql/data postgres

We are starting a new container from the postgres image with a name ‘pg-docker’ with default user ‘postgres’ and our own password ‘start123’. The port 5432 on the host machine is mapped to the port 5432 on the container. The folder/volume ‘$HOME/docker/volumes/postgres’ that we created in the previous step on the host machine is mapped to the ‘/var/lib/postgresql/data’ folder on the container. And using the flag -d we are starting the container in a detached mode.

Verify if the postgres container has started successfully using the below command.

$ docker ps -a

Make a note of the container-id for the postgres from the above command.

Connect to the running container using the below command and create new ROLE, DATABASE and grant all the required permissions for the new user.

Replace the container id with the above-noted value.

$ docker exec -it <container-id> bash
root@1c3102ea09d0:/# psql -U postgres
psql (13.2 (Debian 13.2-1.pgdg100+1))
Type "help" for help.

postgres=# CREATE ROLE teamcityuser with password 'start123' login;
CREATE ROLE
postgres=# create database teamcitydb with owner teamcityuser;
CREATE DATABASE
postgres=# grant all on database teamcitydb to teamcityuser;
GRANT

With all these steps, we are done with the setup required for the Postgres database.

Run the below commands to create the required folders for storing the TeamCity server-specific data and logs.

$ cd ~
$ mkdir teamcity
$ cd teamcity
$ mkdir data
$ mkdir logs
$ cd ~
Run the below command to pull and start the TeamCity server.

$ docker run -d --name teamcity-server -v /home/ec2-user/teamcity/data:/data/teamcity_server/datadir -v /home/ec2-user/teamcity/logs:/opt/teamcity/logs -p 8111:8111 -u 0 jetbrains/teamcity-server

We are starting a new container from the jetbrains/teamcity-server image with a name ‘teamcity-server’ and mapping port 8111 on the host machine to port 8111 on the container. The folders/volumes ‘/home/ec2-user/teamcity/data’ & ‘/home/ec2-user/teamcity/logs’ that we created in the previous step on the host machine are mapped to the ‘/data/teamcity_server/datadir’ & ‘/opt/teamcity/logs’ folders on the container respectively. And using the flag -d we are starting the container in a detached mode with root user using -u 0.

Docker takes some time to pull, extract the contents and start the container. Verify if the teamcity-server container has started successfully using the below command.

$ docker ps -a

Once the server has started, navigate to :8111 on the browser. We get the same screens as discussed in the native installation method. Navigate through all the required steps and proceed with the configuration.

The only difference here is that since we have configured a postgres database, we need to configure it to use it with the TeamCity server. On the database connection setup screen, choose database type as PostgreSQL(step-1). Click on Refresh JDBC drivers(step-2). Enter the database host and port details(step-3). Enter the database name(step-4), username(step-5) and password(step-6) and click on Proceed.

As we have created previously:

Host — machine1 IP address
Port — 5432
Database name — teamcitydb
Database username — teamcityuser
Database password — start123

2. Using Linux Machine 2 To Add An Agent To TeamCity Server

Run the below commands to create the required folders for storing the TeamCity agent-specific data.

$ cd ~
$ mkdir teamcity
$ cd teamcity
$ mkdir data
$ cd ~
Run the below command to pull and start the TeamCity agent.

$ docker run -d --name teamcity-agent1 -e SERVER_URL="teamcityserverhostipaddress:8111" -e AGENT_NAME="agent1" -v /home/ec2-user/teamcity/data:/data/teamcity_agent/conf -v /var/run/docker.sock:/var/run/docker.sock jetbrains/teamcity-agent

Make sure to replace the teamcityserverhostipaddress with the TeamCity server IP address.

We are starting a new container from the jetbrains/teamcity-agent image with a name ‘teamcity-agent1’ and connecting it to the TeamCity server which is running on another machine on port 8111. The folder/volume ‘/home/ec2-user/teamcity/data’ that we created in the previous step on the host machine is mapped to the ‘/data/teamcity_agent/conf’ folder on the container. And using the flag -d we are starting the container in a detached mode with agent name ‘agent1’.

Docker takes some time to pull, extract the contents and start the container. Verify if the teamcity-agent container has started successfully using the below command.

$ docker ps -a

Once the TeamCity server and agent setup are complete, we need to authorize the agent to start using it. Navigate to the server URL on the browser at :8111, click on Agents(step-1), click on Unauthorized tab(step-2) and select the agent and authorize(step-3).

That’s it! We can now start building the pipelines with TeamCity and use them for various jobs.

Accelerate your release velocity with blazing fast test automation cloud!

3. TeamCity Cloud

If you have observed the above two methods of installing TeamCity, we need to configure the server, agents and maintain the infrastructure on our own. Considering this, TeamCity has recently launched TeamCity Cloud, a fully-managed CI/CD solution that automatically runs the builds and tests in the cloud servers. It lets development teams deliver quality software faster and also at scale. This is currently available in the beta phase, and TeamCity is planning for a public release soon.

You can navigate TeamCity Cloud and create your TeamCity instance on the cloud.

Once the request is approved, TeamCity will send a confirmation to the registered email address. As we see below, TeamCity configures the server and adds a couple of agent machines to the cloud instance.

As of now, TeamCity cloud is free based on certain build credits. Once it is officially released to the public, you need to take a subscription for using it. Using TeamCity cloud is one of the easiest ways to explore and learn it without any extra infrastructure setup or provisioning.

Creating Our First Build Configuration

Note: Whatever the TeamCity installation type you have followed, the below steps will be the same and applied for all instances.

TeamCity configurations always start with a project. A project is a collection of all different build configurations. We can create a project in TeamCity corresponding to a software project, specific release/version of a project, or any other logical grouping as per the product. These projects can be nested and organized into a tree-like structure. This will enable the hierarchical display and propagate the settings across projects.

TeamCity creates a Root project at the top of the project hierarchy. This project has most of the similar properties of a usual project. The settings and configurations made in the root project are available to all the other child projects created on the server.

The root project is unique and has the below characteristics:

It is created by default, and we can’t remove it.
It is created as a top-level project; hence it has no parent project associated with it.
It doesn’t have any build configurations.
It does not appear in the normal user-level UI screens as it is primarily present as an entity in Administration UI only.

Let’s start creating a new project by clicking on the Create Project button on the home screen and choose the option Manually. We will give the project details as below and create a simple HelloWorld project.

Once the project is created, we need to create a build configuration under it.

A build configuration in TeamCity is a collection of settings used to start a build or group the sequence of builds in the UI console. Some examples of the build configurations are checking out the code, running the integration tests, building the artifact, etc. Build configuration is specific to a project, and it contains different builds. We can explore all the details of a build configuration on its home page or update it on the edit page.

Click on Create Build Configuration.

Enter the below details and create the build configuration.

When creating a build configuration in TeamCity, we should configure the sequence of steps to be executed. These build steps are configured in the Build Steps section of the Build Configuration Settings page.

We can perform these steps here:

Adding new steps.
Using the auto-detect option in TeamCity to scan the VCS repo and choose the steps.
Copy and delete the steps.
Temporarily enable/disable the build steps.

Click on Add build step, choose the runner type as Command-Line, enter the script below, and finally save.

We are done with configuring our first build configuration in TeamCity for the HelloWorld project. We are good to run our build and see if it’s working. As you have noticed, we are trying to use the echo command to print the sample text to the console.

On the build configuration page — we can see a Run button. Click on the Run button to start the build. TeamCity server starts the build and executes the build steps sequentially on the available agent. Once the build is completed, we can see and verify all the details using different tabs available.

Click on Show full log to see the complete build log.

TeamCity displays the complete build log. As we see in the image, the echoed text is printed to the console.

That’s all! We can use TeamCity for test automation and create CI/CD pipelines for different purposes.

Through this usability testing tutorial, you will learn how usability testing is a great way to discover unexpected bugs, find what is unnecessary or unused before going any further, and have unbiased opinions from an outsider.

Using TeamCity for Running Selenium Test Automation

Now we have got an understanding of installing and configuring TeamCity. Also, we created a project and were able to create build configurations, build steps, and run them. Let us explore creating CI/CD pipelines for running test automation with TeamCity. Selenium Webdriver is a collection of open-source libraries used for testing web applications. The API is used to automate the web application flow and verify if it’s working as expected or not. It supports most common browsers such as Firefox, Chrome, Safari, Edge, and Internet Explorer.

Let us consider the below test case for our example. We use the Selenium Webdriver library to automate one of the UI flows on LambdaTest’s sample website — https://lambdatest.github.io/sample-todo-app/.

We will use WebDriver manager to configure the Firefox driver instance and use it further to interact with the elements of the page. We will also leverage TestNG to control the test execution flow — set up the driver before the test, running the actual test step, and close once the test execution has been completed.

Complete code for the below example is available on Github here for reference.

package com.lambdatest;

import io.github.bonigarcia.wdm.WebDriverManager;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.testng.annotations.AfterTest;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Test;

public class Lambdatest {

   public WebDriver driver;

   @BeforeTest
   public void setUp() {
       WebDriverManager.firefoxdriver().setup();
       driver = new FirefoxDriver();
   }

   @Test
   public void testScript() {
       try {
           driver.get("https://lambdatest.github.io/sample-todo-app/");
           driver.findElement(By.name("li1")).click();
           driver.findElement(By.name("li2")).click();
           driver.findElement(By.id("sampletodotext")).clear();
           driver.findElement(By.id("sampletodotext")).sendKeys("Yey, Let's add it to list");
           driver.findElement(By.id("addbutton")).click();
           driver.quit();
       } catch (Exception e) {
           System.out.println(e.getMessage());
       }
   }

   @AfterTest
   public void teardown() {
       if (driver != null) {
           driver.quit();
       }
   }
}

The complete project structure is as below.

Let us begin with creating a new TeamCity project for running the automated tests in Selenium. Click on Create Project and choose From a repository URL. Since TeamCity needs to pull the code from any repository like GitHub, BitBucket, or Gitlab — we need to configure it using this option.

Enter the below details — repository URL for Github, username, password (if required) and click on Proceed. Username and password are not mandatory and needed only for the repositories, which require authentication for cloning operation.

TeamCity makes a connection to the provided repository and displays the below screen. Proceed with the defaults or choose the required branch name and build a name as needed. In our case, we are proceeding with the master branch.

Upon clicking on Proceed, TeamCity creates a project, build configuration, and VCS root as appropriate as below. TeamCity is also intelligent enough to scan the whole repository and suggest the required build steps automatically.

Since we are using a Java-based project with maven as the build tool, TeamCity has detected that we would like to run the maven commands, specifically the maven goal clean test and pom.xml.

We can either choose the suggested build step by TeamCity or proceed with creating a new one. We would select the recommended step here and click on Use Selected. A new build step would be added to the build configuration as below.

Click on Edit and verify the configurations that TeamCity adds by default.

We can also see the advanced options and configure them as needed. Since in our example, we need to let maven know which suite file to run the tests from, we should pass additional command line parameters. Update the command line parameter as below. The important note here is that we added a build step in the TeamCity build configuration to execute the maven goal clean test using the maven runner. Once everything looks good, click on Save.

Now our build configuration is completed with the required steps to run the build. Before running, let us have a look at the Version Control Settings option. Click on Version Control Settings in the left pane. The configuration looks as below at the moment.

As we see above in the Version Control Settings pane, a new VCS Root has been added to the build configuration. Whenever TeamCity needs to get the source code, it needs to make a connection to the version control system(VCS), which is referred to as a VCS root. TeamCity monitors the changes in VCS and gets the sources for a specific build configuration. All these configuration parameters, such as the path to the VCS repository source, user name, password, and any additional settings, are represented by the VCS root only.

VCS roots are created within a project. Once created they are available to all the build configurations added to that project and its child projects. We can configure more than VCS root to a build configuration. We can even configure additional options using the Show Advanced Options; also, we can create custom checkout rules.

Click on the Run button to trigger and execute the build steps.

TeamCity starts pulling the latest code from the VCS root and specific branches and executing the build steps one by one. In our case, there is only one step to run the maven goal. So it starts pulling the required dependencies to build the project and starts the tests. Once all the build steps are executed, we see the below screen with the overall status.

Let us click on the Show Full log option to see the whole execution log for the run.

As we see below, the expected maven command is executed, and the tests are completed.

mvn clean install -DsuiteXmlFile=testnglocal.xml

This is great! We can run a Selenium Automated test in our local machine using the build configuration that we have just created in TeamCity. We could also leverage some of the below additional features of TeamCity for test automation and develop robust TeamCity build pipelines.

Triggers:

Triggers are used to add builds to the queue. We can add a trigger when an event occurs (for ex. VCS check-in) or periodically with some configurable interval. We could use a schedule trigger and configure to run the build periodically — this is used while configuring the test suite for nightly regression execution.

Parameters:

Parameters in TeamCity enable the users to share different settings with the running build. These are key-value pairs, which can be defined by the user or provided by TeamCity.

There are three types of build parameters:

Environment variables are passed into the generated build process as environment and defined with the env. prefix.
System properties are passed into the build scripts of the TeamCity’s supported runners as variables and are defined with the system. prefix. They are specific to a build tool.
Configuration parameters are not passed into the build but are only supposed to share the settings within a build configuration. These are useful for customizing a build configuration based on a template. They don’t contain any prefixes.

Failure conditions

We can define the conditions under which TeamCity marks a build as failed. These are some of the available conditions.

Build Features

A building feature is an additional functionality that we can add to TeamCity’s build configuration. It affects the running builds and results. We could add a build feature to process the XML reports generated by TestNG after the test execution is over.

Statistics

When we have multiple projects and several build configurations, tracking all those statistics is very important. TeamCity collects all the statistical data across builds and presents a visual chart for easy analysis.

DSL

Besides storing the settings in a version-controlled system, TeamCity also allows storing the settings as a Kotlin DSL(Domain Specific Language). Using the DSL configuration stored in version control systems enables you to define the settings programmatically. Since Kotlin is a statically typed language, you automatically receive the auto-completion feature in an IDE, discovering available API options much more straightforward.

Coming back to our example, we have executed our tests in only one browser — firefox. What if we need to run the tests in other browsers as well? And we might have some of the below requirements from the TeamCity test automation perspective.

Save the test execution reports and logs over some time.
Able to spin up different browsers including multiple versions i.e. Cross-Browser Testing.
Integrate the test framework with other tools like Jira or GitHub, or Trello for bug creation on failure.
Support a vast number of tests — thousands of tests across different browsers, OS, and resolutions.

To achieve all this, we need a solution that caters to all the requirements and helps us do continuous testing, providing early feedback to the development team. LambdaTest is one such solution to all these requirements. In the next section, we will integrate our example with the LambdaTest cloud Selenium grid and perform the test automation using TeamCity.

Integration With LambdaTest Cloud-Based Selenium Grid

LambdaTest is a cross-browser testing cloud solution to perform live interactive and automated cross-browser testing on 3000+ real browsers and operating systems online. It is one of the most often used Selenium grid solutions for many companies.

To get started with the LambdaTest cloud, navigate to LambdaTest Register Page and sign-up for a new account. Once your LambdaTest account is activated, go to your Profile section and note the username and access key. We would need these LambdaTest username and access key to execute our tests on the LambdaTest platform.

Before using LambdaTest with our TeamCity build pipeline, we need to download and install the plugin from the marketplace.

Visit the JetBrains marketplace and download the “LambdaTest” plugin.
To install the downloaded TeamCity plugin, follow the below steps to upload the plugin on your TeamCity server.
Go to your server Administration and then to Plugins List.
Choose the option Upload plugin zip in the Plugins screen.
Choose the zip file and upload it to TeamCity.
Once uploaded, click on Enable uploaded plugins.

At this moment, we can see the LambdaTest plugin added to the External Plugins as below.

Now go to the existing Selenium test build configuration, and update the required testng XML file in the Additional Maven command line parameters.

Click on Build Features and then click on Add build feature. Select LambdaTest from the dropdown list.

Enter the username, and access token noted as above and save. The build feature for LambdaTest is added now, and these values/parameters are passed to our test case once the build is started.

Run the build. Once the build has started, navigate to the LambdaTest Automation dashboard and verify the execution status.

LambdaTest also provides all the test logs and recordings of the test execution as below. It will be very helpful for any issues with the script and using logs we can debug and identify the root cause of the issue.

In the above example, we have executed our tests on the Chrome browser on Windows OS.

LambdaTest also provides the desired capabilities generator to help us with the required configuration values to run the tests on different platform combinations. It is a valuable utility and comes in handy while working with cross-browser testing.

A comprehensive User Acceptance Testing (UAT) tutorial that covers what User acceptance testing is, its importance, benefits, and how to perform it with real-time examples.

Conclusion

We have now successfully learned how to create a CI/CD pipeline using TeamCity and performed Selenium test automation on the LambdaTest cloud grid. We have covered some of the essential concepts used in TeamCity that help build pipelines and use them for test automation.

There are numerous CI/CD benefits and among them, TeamCity is a robust CI/CD solution to manage the pipelines, and in turn, helps the teams to follow best practices of Continuous Testing and other DevOps principles. We can even integrate IDE like IntelliJ with TeamCity plugins and start running the jobs from the IDE itself. Teams can build robust pipelines using TeamCity and leverage LambdaTest cloud grid solution for achieving excellent test automation benefits.

Happy Building and Testing!

Building A CI/CD Pipeline With Travis CI, Docker, And LambdaTest

RakeshVardhan1 — Wed, 23 Nov 2022 16:49:25 +0000

With the help of well-designed Continuous Integration systems in place, teams can build quality software by developing and verifying it in smaller increments. Continuous Integration (CI) is the process of pushing small sets of code changes frequently to the common integration branch rather than merging all the changes at once. This avoids big-bang integration before a product release. Test automation and Continuous Integration are an integral part of the software development life cycle. As the benefits of following DevOps methodology in a development team becomes significant, teams have started using different tools and libraries like Travis CI with Docker to accomplish this activity.

In this blog, we will discuss the role of Travis CI in Selenium test automation and dive deep into using Travis CI with Docker. We will also look at some effective Travis CI and Docker examples. We would also integrate and automate Selenium tests suites with Travis CI on the LambdaTest cloud grid.

Without further ado, Let’s build a CI/CD pipeline with Travis CI, Docker, and LambdaTest.

Perform browser test automation testing on the most powerful cloud infrastructure. Leverage LambdaTest automation testing for faster, reliable and scalable experience on cloud.

Overview Of Travis CI

Travis CI is a cloud-based service available for teams for building and testing applications. As a continuous integration platform, Travis CI supports the development process by automatically building and testing the code changes, providing immediate feedback if the change is successful. It can also help us automate other steps in the software development cycle, such as managing the deployments and notifications.

Travis CI can build and test projects on the cloud platform hosted on code repositories like GitHub. We could also use other best CI/CD tools such as Bitbucket, GitLab, and Assembla, but some of them are still in the beta phase. When we run a build, Travis CI clones the repository into a brand-new virtual environment and performs a series of steps to build and test the code.

When a build is triggered, Travis CI clones the GitHub repository into a brand-new virtual environment and performs a series of tasks to build and test the code. If any of those jobs fails, the build is considered broken; else, the build is successful. On the success of the build, Travis CI will deploy the code to a web server or an application host.

In case you are eager to learn about the Travis CI/CD pipeline, please refer to our detailed blog that deep dives into how to build your first CI/CD pipeline with Travis CI.

.travis.yml Configuration

Builds on Travis CI are configured mostly through the build configuration defined in the file .travis.yml in the code repository. It allows the configuration to be version-controlled and flexible. Once the application code is completed, we need to add the .travis.yml file to the repository.

It contains the instructions on what to build and how exactly to test the application. Travis CI performs all the steps configured in this YAML file.

Some of the commonly used terms associated with Travis CI are:

Job — Job is an automated process that clones the code repository into a brand-new virtual environment and performs a set of actions such as compiling the code, running tests, deploying the artifact, etc.
Phase — Job is a collection of different phases. Sequential steps, aka phases, collectively create a job in Travis CI. For example, install phase, script phase, before_install phase, etc.
Build — Build refers to a group of jobs that are running in a sequence. For example, a build can have two jobs defined in it. Each job tests the project with a different version of the programming language. The build is finished only when all of its jobs have completed execution.
Stage — Stage refers to the group of jobs that run in parallel.

Features Of Travis CI

Some of the salient features that Travis CI provides as a CI/CD platform are given below:

Free cloud-based hosting
Automatic integration with GitHub
Safelisting or Blocklisting branches
Pre-installed tools and libraries for build and test
Provision to add build configuration via a shell script
Caching the dependencies
Building Pull-Requests
Support for multiple programming languages.
Build Configuration Validation.
Easily set up CRON jobs.

In case you want to deep dive into how Jenkins (the preferred open-source CI/CD tool) stacks up against Travis CI, please refer to our blog Travis CI Vs. Jenkins to make an informed decision.

Accelerate your release velocity with blazing fast automated testing on cloud!

Overview Of Docker

According to the Stack overFlow 2021 Developer survey, Docker is one of the most used containerization platforms to develop, ship, and run applications.

Docker enables us to separate applications from the infrastructure so that the application software can be delivered quickly. In short, Virtual Machines(VM) virtualizes the hardware, whereas Docker virtualizes the Operating System(OS).

In a Virtual Machine — multiple operating systems are run on a single host machine using the virtualization technology known as Hypervisor. Since the full OS needs to be loaded, there may be a delay while booting up the machines; also, there is an overhead of installing and maintaining the operating systems. Whereas in Docker, the host operating system itself is leveraged for virtualization.

Docker daemon, installed on the host machine, manages all the heavy lifting tasks like listening to the API requests, managing docker objects (images, containers, and volumes), etc. In case you are looking to leverage Docker with Selenium, do check out our detailed blog that outlines how to run Selenium tests in Docker.

Some of the useful concepts Docker leverages from the Linux world are as follows:

Namespaces — Docker uses a technology called namespaces to provide the isolated workspace called container. For each container, Docker creates a set of namespaces when we launch it. Some namespaces in Linux are pid, net, ipc, mnt, and uts.
Control Groups — Docker Engine on Linux also uses another technology known as control groups (cgroups). A cgroup restricts an application to a limited set of resources.
Union file systems — Union file systems, or UnionFS, are lightweight and fast file systems that work by creating layers. Docker Engine makes use of UnionFS to provide container building blocks.
Container Format — Docker Engine combines the namespaces, control groups, and UnionFS into a container format wrapper. The default container format is libcontainer.

Basics Of Docker

Docker uses a client-server architecture at its core. Some of the common objects and terminology used in Docker are explained below.

Image — An image is a read-only template with all the required instructions to create a Docker container. The image is a collection of files and metadata. These files collectively form the root filesystem for the container. Typically images are made up of layers which are stacked on top of each other. And these images can share layers to optimize disk usage, memory usage, and transfer times.
Container — Container is the runnable instance of an image. Containers have their lifecycle to create, start, stop, delete, and move using the Docker client API commands. Containers are by default isolated from other containers and the host machine. Basically, the container is defined by the image it is created from. If required, we need to give specific configuration or environment variables before starting it.
Engine — Docker Engine is a client-server application that includes the following major components.
A server is a form of long-running application known as a daemon process (i.e. the Dockerd command).
A REST API defines interfaces that applications can use to communicate with the daemon and instruct it on what needs to be done.
A client for the command-line interface (CLI) (i.e. the Docker command).
Registry — Registry is a place where all the Docker images are stored and distributed. Docker hub manages all the openly available docker images.
Network — Docker uses a networking subsystem on the configurable host OS, using different drivers. These drivers provide the core networking functionality.
Bridge
Host
Overlay
Macvlan
none
Volume — Since Docker containers are ephemeral by default, we need a provision to store the data between 2 different containers. Docker volumes provide this functionality where the data is persisted across containers.
Dockerfile — It is a flat-file with instructions on how to create an image and run it.
Docker-compose — Docker compose is a utility tool for defining and running multi-container docker applications. Here we use a YAML file to configure the services required for the application. By using a single command, we can start all the services and also establish dependencies between them.

This article explains the emulator vs simulator vs real device differences, the learning of which can help you select the right mobile testing solution for your business.

How to Integrate Travis CI with Selenium using Docker

Using Docker Images

Now that we have understood the fundamentals of Travis CI and Docker let us start using them together to run some UI tests with Selenium WebDriver.

The first step is to define a test scenario for our Travis CI and Docker Example. Here is the Selenium test scenario:

Open a specific browser — Chrome, Firefox, or Edge
Navigate to the sample application
Verify headers on the page
Select the first two checkboxes and see if they are selected
Clear the textbox — Want to add more and enter some text into it and click on Add
Check if the new item is added to the list and verify its text

Here is the code that we need to write using Selenium WebDriver agnostic of any browser. First, we use the BrowserFactory to instantiate the browser based on the input and pass it on to our tests.

package com.lambdatest;

import org.openqa.selenium.By;
import org.openqa.selenium.WebElement;
import org.testng.annotations.AfterTest;
import org.testng.annotations.BeforeTest;
import org.testng.annotations.Test;

import static org.assertj.core.api.Assertions.assertThat;

public class SeleniumTests extends BaseTest {

    @BeforeTest
    public void setUp() {
        driver.get("https://lambdatest.github.io/sample-todo-app/");
    }

    @Test(priority = 1)
    public void verifyHeader1() {
        String headerText = driver.findElement(By.xpath("//h2")).getText();
        assertThat(headerText).isEqualTo("LambdaTest Sample App");
    }

    @Test(priority = 2)
    public void verifyHeader2() {
        String text = driver.findElement(By.xpath("//h2/following-sibling::div/span")).getText();
        assertThat(text).isEqualTo("5 of 5 remaining");
    }

    @Test(priority = 3)
    public void verifyFirstElementBehavior() {
        WebElement firstElementText = driver.findElement(By.xpath("//input[@name='li1']/following-sibling::span[@class='done-false']"));
        assertThat(firstElementText.isDisplayed()).isTrue();
        assertThat(firstElementText.getText()).isEqualTo("First Item");

        assertThat(driver.findElement(By.name("li1")).isSelected()).isFalse();
        driver.findElement(By.name("li1")).click();
        assertThat(driver.findElement(By.name("li1")).isSelected()).isTrue();

        WebElement firstItemPostClick = driver.findElement(By.xpath("//input[@name='li1']/following-sibling::span[@class='done-true']"));
        assertThat(firstItemPostClick.isDisplayed()).isTrue();

        String text = driver.findElement(By.xpath("//h2/following-sibling::div/span")).getText();
        assertThat(text).isEqualTo("4 of 5 remaining");
    }

    @Test(priority = 4)
    public void verifySecondElementBehavior() {
        WebElement secondElementText = driver.findElement(By.xpath("//input[@name='li2']/following-sibling::span[@class='done-false']"));
        assertThat(secondElementText.isDisplayed()).isTrue();
        assertThat(secondElementText.getText()).isEqualTo("Second Item");

        assertThat(driver.findElement(By.name("li2")).isSelected()).isFalse();
        driver.findElement(By.name("li2")).click();
        assertThat(driver.findElement(By.name("li2")).isSelected()).isTrue();

        WebElement secondItemPostClick = driver.findElement(By.xpath("//input[@name='li2']/following-sibling::span[@class='done-true']"));
        assertThat(secondItemPostClick.isDisplayed()).isTrue();

        String text = driver.findElement(By.xpath("//h2/following-sibling::div/span")).getText();
        assertThat(text).isEqualTo("3 of 5 remaining");
    }

    @Test(priority = 5)
    public void verifyAddButtonBehavior() {
        driver.findElement(By.id("sampletodotext")).clear();
        driver.findElement(By.id("sampletodotext")).sendKeys("Yey, Let's add it to list");
        driver.findElement(By.id("addbutton")).click();
        WebElement element = driver.findElement(By.xpath("//input[@name='li6']/following-sibling::span[@class='done-false']"));
        assertThat(element.isDisplayed()).isTrue();
        assertThat(element.getText()).isEqualTo("Yey, Let's add it to list");
    }

    @AfterTest
    public void teardown() {
        if (driver != null) {
            driver.quit();
        }
    }
}

We are using TestNG and adding appropriate annotations for our tests. We also configured the testng.xml file to include what tests to run.

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="All Test Suite">
    <test verbose="2" name="travisci-selenium-docker-lambdatest">
        <parameter name="browser" value="GRID_CHROME">
            <parameter name="gridHubURL"
                       value="http://localhost:4444/wd/hub"/>
        </parameter>
        <classes>
            <class name="com.lambdatest.SeleniumTests">
                <methods>
                    <include name="verifyHeader1"/>
                    <include name="verifyHeader2"/>
                    <include name="verifyFirstElementBehavior"/>
                    <include name="verifySecondElementBehavior"/>
                    <include name="verifyAddButtonBehavior"/>
                </methods>
            </class>
        </classes>
    </test>
</suite>

And finally, we use the below Maven command to trigger the tests. In case you are relatively new to Maven, do check out our detailed blog that will help you getting started with Maven for Selenium Automation.

mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_FIREFOX -DgridHubURL=http://localhost:4444/wd/hub

The entire code for this project is available on the GitHub repository.

Since our goal is to configure and run the Selenium tests using Travis CI and Docker, you should check the following prerequisites:

Active GitHub account, with the code repository available in GitHub.
Active TravisCI account and the required permissions to access the repository from GitHub.

Once all these prerequisites are met, let’s take a quick look at the travis.yml file configured in the project.

dist: trusty
language: java

jdk:
  - oraclejdk8

before_script:
  - docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-firefox:4.0.0-rc-1-prerelease-20210618
  - docker ps

script:
  - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_FIREFOX -DgridHubURL=http://localhost:4444/wd/hub

cache:
  directories:
    - .autoconf
    - $HOME/.m2

deploy:
  provider: releases
  api_key: ${api_key}
  skip_cleanup: true
  file: [ "target/surefire-reports/emailable-report.html",
          "target/surefire-reports/index.html" ]
  on:
    all_branches: true
    tags: false

As we learned earlier, travis.yml is the configuration file that we use to instruct the Travis CI cloud server on what actions to perform for the given project.

Let’s understand all these details one by one.

dist: When a new job is started in Travis CI, it spins up a new virtual environment in the cloud infrastructure. Each build is run in the pre-configured virtualization environments. Then, using the dist key, we specify the infrastructure needed to be spun up by the server. Some of the available values for dist in the Linux environment are trusty, precise, bionic, etc.

language: Language key is used to specify the language support needed during the build process. We can choose the language key appropriately from the list of available values. Some examples include Java, Ruby, Go, etc.

jdk: The key JDK is used while the language key is selected as Java. We basically give the version of the JDK that needs to be used while building the project. In our case, it should be oraclejdk8 as we are using Java 1.8 version.

script: Script key is used to run the actual build command or script specific to the selected language or environment. In the example, we use mvn commands as we have created a maven-based project for the example.

before_script: It is used to run any commands before running the actual script commands. Before running the Maven command in the script phase, we need to ensure the Docker environment is set up successfully.

cache: This key is used to cache the content that doesn’t change frequently, and running this can help speed up the build process. Cache has many supported keys available for different caching strategies. Some of them are directories, npm, cache, etc.

directories: This is a caching strategy used for caching the directories based on the given string path.

We can also use the build configure explorer from Travis CI and check how the Travis CI system reads and understands the configuration. This is helpful in validating the configuration, adding the correct keys, and updating as per the latest specifications.

Once the prerequisites mentioned above are met, we can start with our build-in Travis CI with Docker. There are various ways in which we can start the builds.

Commit the code and push to Github; the default hook will trigger a new build on the server.
Manually trigger a build from the dashboard views.
Trigger with the Travis CI API.

Here is the default dashboard view of my user in Travis CI. As we can see, I have activated only two projects from GitHub.

From any of the options described above, Travis CI will start performing the below actions when a build is triggered:

Reads the configuration defined in travis.yml
Adds the build to the job queue
Start creating the brand new virtual environment to execute the steps mentioned in the travis.yml file

In the View config tab, we can also see the build configuration as read by Travis CI. Since we haven’t specified any os value in the travis.yml file, it takes the default value as Linux and proceeds with the build set-up.

Once the build is completed, we could see the image banner which indicates that the build has passed successfully. It also gives us other information such as branch information, time taken for execution, latest commit message, operating system information, build environment, etc.

Let’s navigate to the Job log section and understand what Travis CI and Docker have performed for our build.

In the Job log section, we can see the complete log information for the current job. As seen below, Travis CI has spun up the below worker for our build. It has created a new instance in the Google Cloud Platform, and the start-up time is around 6.33 seconds.

Travis CI will dynamically decide which platform (like GCP Compute Engine or AWS EC2 machines) to choose for the instance. If an enterprise version is used, we could also configure our infrastructure to be used by Travis CI.

In the Build system information from the logs, we could identify the build language used, the distribution of the OS, Kernel version, etc. For each operating system, Travis CI has a set of default tools defined already. Travis CI will install and configure all the required services required for the build. In this case, Git is installed.

Also, the Docker client and server will be installed by default. We can verify the version used by these components in the below screenshot:

Since our build config uses Java JDK, Travis CI will configure the JDK as specified in the travis.yml file and update the JAVA_HOME path variable as appropriate.

Next, it clones the latest code from the specific branch(master in our example) and updates the current directory. It also reads the environment variables if defined and updates the set-up as needed. Finally, it also checks if there is any build-cache defined for the project. We will discuss more about build cache in the later section.

Before pointing our tests to a remote server, we should have a browser configured on a specific port and listen for the incoming requests. From our example, we have configured the RemoteWebDriver as below for launching our tests in Chrome.

case GRID_CHROME:
                ChromeOptions chromeOptions = new ChromeOptions();
                chromeOptions.setCapability(CapabilityType.ACCEPT_SSL_CERTS, true);
                try {
                    driver = new RemoteWebDriver(new URL(gridHubURL), chromeOptions);
                } catch (MalformedURLException e) {
                    logger.error("Grid Hub URL is malformed: {}", e.getMessage());
                }
                break;

If you remember the travis.yml file in our project, we have a before_script stage defined, and we have already added the below two commands.

- docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-firefox:4.0.0-rc-1-prerelease-20210618
- docker ps

Let’s understand the commands that we have used.

The first one is the Docker run command that starts the Selenium Firefox container on port 4444.

docker: Docker is the base command for its CLI

run: Runs the commands in a new container

-d: Command option for Docker run to run the container in the background

-p: Command option for Docker run to publish a container’s port to the host port. Here we are mapping the 4444 port of the container to the 4444 port in the host. The format is host port: container port

-v: Command option for Docker run to bind mount a volume. Here we are using the shared memory /dev/shm in the container and mapping it to the shared memory /dev/shm of the host.

Next is the image that we are using to run the docker command. Here we are using the Selenium Firefox standalone image with tag 4.0.0-rc-1-prerelease-20210618. More information about the latest versions available can be found from the official Selenium Github repository and the Docker hub public registry.

Using the below command, Travis CI pulls the specified Selenium Firefox image from the Docker registry and starts creating a new container with the provided configuration. Once it has successfully created a container, we can see the list of the running containers using the second command:

docker ps

As we observe from the job logs, the docker run has been executed successfully, and the running container has been listed with the container ID, image name used, ports mapped between container and host, volumes mounted, and status of the container.

With this, the required setup is completed for running our tests with Travis CI and Docker. The next step is to start sending the HTTP requests to this container on a specific port. We do this using the Maven command as shown below.

mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_FIREFOX -DgridHubURL=http://localhost:4444/wd/hub

Here we are using the Maven build command along with TestNG to clean the repository and using the install phase with the command arguments such as the suite file, required browser to use, and the grid URL.

Since the Firefox container is running in the Travis CI instance on port 4444, we send the test requests to this port on the mentioned URL.

Once the Maven build is started, Travis CI downloads the required dependencies from the Maven central repository for the first time. Since we are using the cache configuration for Maven as below, it uses the cache for subsequent runs.

Therefore, Travis CI can cache the content for the build so that the build process can be sped up. In order to use this caching feature, we need to set the Build pushed branches to ON in the repository settings.

cache:
  directories:
    - .autoconf
    - $HOME/.m2

Once all the tests are executed on the browser which is available in the Docker container, we get the results and the build status as zero(0).

We have successfully performed our first Selenium test automation with Travis CI and Docker. If the build completes with an exit code as zero(0), it will be treated as passed. For many of us in Selenium test automation, it is required to see the test results to assess the regression suite quality. So we just added a phase in the travis.yml file to deploy the artifacts back to GitHub once the build is completed.

Since we have used TestNG in our Travis CI and Docker example, we get test execution reports in a clean and precise format in the target folder.

We are trying to deploy the index.html and emailable-report.html as per the below details.

deploy:
  provider: releases
  api_key: ${api_key}
  skip_cleanup: true
  file: [ "target/surefire-reports/emailable-report.html",
          "target/surefire-reports/index.html" ]
  on:
    all_branches: true
    tags: false

Deploy phase details from the Travis CI build logs.

In order to publish the artifacts to GitHub, we need to create a personal access token from the GitHub developer settings page and add it to the environment variables section of the Travis CI repository. The same variable api_key is used in the deploy phase of our configuration.

That’s all! We are done with running one complete build in Travis CI with Docker. In the Build History section, we can see the whole history of test runs as below.

We can see the deployed artifacts in the Releases section on GitHub as below.

We can also run the tests in other browsers using the other browser containers. But, first, we need to update the before_install phase with the relevant run command in the travis.yml file.

before_install:
    - docker run -d -p 4445:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:4.0.0-rc-1-prerelease-20210618
    - docker run -d -p 4446:4444 -v /dev/shm:/dev/shm selenium/standalone-firefox:4.0.0-rc-1-prerelease-20210618
 - docker run -d -p 4447:4444 -v /dev/shm:/dev/shm selenium/standalone-edge:4.0.0-rc-1-prerelease-20210713
    - docker ps

Also need to update the -DgridHubURL in the maven command with a configured host port for each browser.

A complete webdriver tutorial that covers what WebDriver is, its features, architecture and best practices.

Using Docker Compose

In our previous Travis CI and Docker example, we have used Docker standalone images from Selenium and started running our tests in Travis CI with Docker. However, if you would have observed, adding each Docker run command in the travis.yml file is not fool-proof, and we need to update the travis.yml file for all the changes required on the browser end.

Instead, we could leverage the docker-compose utility tool from Docker itself. We can define all the browser-specific services required in a file called docker-compose.yml file and refer to it in the travis.yml file. It would really help us improve the readability of the build configuration file and separate the browser containers.

Using docker-compose we can use a single command to activate all the services in a single go. Also, docker-compose files are very easy to write as they also use the YAML syntax. All the services in docker compose can even be stopped with a single command.

We use the below docker-compose.yml file to define the services.

# To execute this docker-compose yml file use `docker-compose -f docker-compose-v3.yml up`
# Add the `-d` flag at the end for detached execution
# To stop the execution, hit Ctrl+C, and then `docker-compose -f docker-compose-v3.yml down`
version: "3"
services:
  chrome:
    image: selenium/node-chrome:4.0.0-rc-1-prerelease-20210618
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6900:5900"

  edge:
    image: selenium/node-edge:4.0.0-rc-1-prerelease-20210618
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6901:5900"

  firefox:
    image: selenium/node-firefox:4.0.0-rc-1-prerelease-20210618
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6902:5900"

  selenium-hub:
    image: selenium/hub:4.0.0-rc-1-prerelease-20210618
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"

So now the new travis.yml configuration looks as below. Update the travis.yml file in the project with this file, and commit and push the code to start the new build in Travis CI and Docker using the new build configuration.

dist: trusty
language: java

jdk:
  - oraclejdk8

before_script:
  - curl https://gist.githubusercontent.com/rakesh-vardan/c1dcf6531b826fad91f18c285d566a71/raw/ad90a18fe5e70f2d6aea06621b76d6e0329a4aab/docker-compose-sel.yml > docker-compose.yaml
  - docker-compose up -d
  - docker-compose ps

script:
  - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_FIREFOX -DgridHubURL=http://localhost:4444/wd/hub

cache:
  directories:
    - .autoconf
    - $HOME/.m2

before_script:
  - curl https://gist.githubusercontent.com/rakesh-vardan/c1dcf6531b826fad91f18c285d566a71/raw/ad90a18fe5e70f2d6aea06621b76d6e0329a4aab/docker-compose-sel.yml > docker-compose.yaml
  - docker-compose up -d
  - docker-compose ps

The only change here is that in the before_script phase, we are using docker-compose instead of running the direct docker commands. Instead, we use the curl utility to download the docker-compose file and save the content to a file named docker-compose.yml.

Once that is completed, we use the docker-compose up -d statement to start all the services. This command starts all the services in the compose file and starts listening to the requests on the mentioned ports. Then we use the docker-compose ps command to see the services that are started. Finally, we use the same maven command to point our tests and execute them using the Travis CI build job.

The below is the screenshot of the Travis CI logs, which shows that the docker-compose has successfully started the defined services.

Build log details while running the tests on the Chrome browser with the below maven command.

mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_CHROME -DgridHubURL=http://localhost:4444/wd/hub

Build log details while running the tests on the Firefox browser with the below maven command.

mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_FIREFOX -DgridHubURL=http://localhost:4444/wd/hub

As you have observed, we are using the same port for running the tests in Chrome and Firefox browsers. It is because we are leveraging the hub-node architecture of the Selenium Grid here. In our earlier Travis CI and Docker example, where we are running the stand-alone container, we haven’t used the hub container.

To run the tests with Travis CI and Docker on Selenium Grid 4 architecture, please read our blog on Selenium Grid 4 Tutorial For Distributed Testing.

Perform browser automation testing on the most powerful cloud infrastructure. Leverage LambdaTest automated testing for faster, reliable and scalable experience on cloud.

Run Selenium Tests With Zalenium And Travis CI

As per our previous examples of using Selenium, Travis CI, Docker images to run the tests, we can clearly see that we always need to update the versions as and when a new version is out. This becomes an overhead, as we need to spend a lot of time updating the configurations.

If you search for an alternative, you might come across a solution called Zalenium. Zalenium is a flexible and scalable container-based Selenium Grid with video recording, live preview, basic auth & dashboard. It provides all the latest versions of browser images, drivers, and tools required for Selenium test automation. It also has a provision to send the test requests to a third-party cloud provider if the browser requested is not available in the setup.

Let’s see how we can use them together using Docker images and with Docker compose.

Using Docker Images

In order to use Zalenium, we need to pull the two images as specified below. Once those images are available, start the container using the Docker run command with the necessary options.

Use the below travis.yml file to configure the build using Docker images for Zalenium.

dist: trusty
language: java

jdk:
  - oraclejdk8

before_script:
  - docker pull elgalu/selenium
  - docker pull dosel/zalenium
  - docker run -d --name zalenium -p 4444:4444 -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/videos:/home/seluser/videos --privileged dosel/zalenium start
  - docker ps

script:
  - mvn clean install -DsuiteXmlFile=${suiteXmlFile} -Dbrowser=${browser} -DgridHubURL=${gridHubURL}

cache:
  directories:
    - .autoconf
    - $HOME/.m2

Also, we have made use of environment variables options available in Travis CI to dynamically pass the values at run time for TestNG XML file name, browser, and grid hub URL. We can set these values as below in the Settings section of the project.

Once the execution is started, we can observe in the logs that the Travis CI system has read the environment variables and passed them to the running build.

Next, the required Docker images are pulled from the central registry, and the Docker run command runs. Finally, the command docker ps shows the running containers at the moment.

The required browser container is started on port 4444, and the tests will be executed successfully. We can see the test execution status as below.

Similarly, we can change the browser value to GRID_FIREFOX and trigger a new build.

Once the execution for GRID_FIREFOX is started, the Travis CI system will read the environment variables and pass them to the running build.

The required browser container is started on port 4444, and the tests will be executed successfully. We can see the test execution status as below.

Using Docker Compose

We can also leverage Docker compose and Zalenium here to quickly spin up a scalable container-based Selenium Grid and run our tests.

The below is the docker-compose.yml file with the required services. Here we are starting two services, Selenium and Zalenium, and listening to the incoming requests on port 4444.

# Usage:
#   docker-compose up --force-recreate
version: '2.1'

services:
  #--------------#

  dep:
    image: "elgalu/selenium"
    container_name: selenium
    command: echo 0
    restart: "no"

  zalenium:
    image: "dosel/zalenium"
    container_name: zalenium
    hostname: zalenium
    tty: true
    volumes:
      - /tmp/videos:/home/seluser/videos
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/bin/docker:/usr/bin/docker
    ports:
      - 4444:4444
    depends_on:
      - dep
    command: >
      start --desiredContainers 2
            --maxDockerSeleniumContainers 8
            --screenWidth 800 --screenHeight 600
            --timeZone "Europe/Berlin"
            --videoRecordingEnabled true
            --lambdaTestEnabled false
            --startTunnel false
    environment:
      - HOST_UID
      - HOST_GID
      - PULL_SELENIUM_IMAGE=true

The complete travis.yml file, in this case, is as below.

dist: trusty
language: java

jdk:
  - oraclejdk8

before_script:
  - curl https://gist.githubusercontent.com/rakesh-vardan/75ab725c9e907772eacb7396383ecd84/raw/4ab799e94cdb9eeaca63528eda97e2d1990f5bc7/docker-compose-zal.yml > docker-compose.yml
  - docker-compose up -d --force-recreate
  - docker-compose ps

script:
  - mvn clean install -DsuiteXmlFile=${suiteXmlFile} -Dbrowser=${browser} -DgridHubURL=${gridHubURL}

cache:
  directories:
    - .autoconf
    - $HOME/.m2

Once the build has started, we can see the log details as below. After that, required services will be started, and the tests are executed.

Run Selenium Tests With Selenium Cloud using Travis CI

From all our previous examples, it is evident that we can use Travis CI, Docker, and Selenium together to create automation regression pipelines with ease. As per our requirement, we can configure the declarative YAML files — travis.yml & docker-compose.yml as per our requirement, trigger the build, and get the results. However, there is one drawback with all our previous examples.

Let’s say if we want to see the regression execution live as it is happening, it is not possible, at least in the community version of Travis CI. On the enterprise version, along with some additional configuration, we could achieve this. Also, if we wanted to check the logs — driver logs, network logs, etc.-. we have to make changes to our examples that required additional effort again. Is there a solution to this problem? The answer is YES! We have LambdaTest cloud-based Selenium grid to our rescue.

LambdaTest is a cross browser testing cloud platform to perform automation testing on 3000+ combinations of real browsers and operating systems. It is very easy to configure a project using LambdaTest and start leveraging the benefits out of it. Let’s integrate our project examples with the LambdaTest cloud grid and see how it’s working.

We try to understand both options — using Docker images and starting the services using the docker-compose.

Using Docker Images

To start with the LambdaTest, first, navigate to the LambdaTest registration page and sign-up for a new account. Once the LambdaTest account is activated, go to the Profile> section and note the LambdaTest Username and LambdaTest Access key. We will need these LambdaTest Username and Access keys to execute our tests on the LambdaTest platform.

We use the below travis.yml file to run our tests with Travis CI and Docker on the LambdaTest cloud grid.

dist: trusty
language: java

jdk:
  - oraclejdk8

before_script:
  - docker pull elgalu/selenium
  - docker pull dosel/zalenium
  - docker run -d --name zalenium -p 4444:4444 -e LT_USERNAME=${LT_USERNAME} -e LT_ACCESS_KEY=${LT_ACCESS_KEY} -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/videos:/home/seluser/videos --privileged dosel/zalenium start --lambdaTestEnabled true
  - docker ps

script:
  - mvn clean install -DsuiteXmlFile=${suiteXmlFile} -Dbrowser=${browser} -DgridHubURL=${gridHubURL}

cache:
  directories:
    - .autoconf
    - $HOME/.m2

Before triggering the build, we need to ensure that the below environment variables are set in the project settings in Travis CI.

LT_USERNAME - Username from the LambdaTest profile page.

LT_ACCESS_KEY - Access key obtained from the LambdaTest profile page.

Below should be the form of gridHubURL.

https://<username>:<accessKey>@hub.lambdatest.com/wd/hub

Where is the Username, is the Access key from LambdaTest account.

Once the build starts, Travis CI reads the configured environment variables.

As per the travis.yml file configuration, the respective Docker images are pulled, and containers are started.

We are using the Docker run command with the flag lambdatestEnabled as true. Because of this, Zalenium starts the hub on port 4444 and one custom node using docker-selenium. A cloud proxy node is registered to the grid since we enabled the cloud integration with the LambdaTest platform. Once the test request is received, it will be sent to the available node with all the capabilities to execute the tests.

Once Travis CI completes the Selenium test automation execution, we can navigate to the Automation dashboard of LambdaTest to see our test execution in the Timeline view. Click on the test case; we would be navigated to the Automation logs screen to see other details such as execution recording, logs, etc.

Next, let’s do the same using the docker-compose.

Refer to our documentation for more details on Zalenium integration with LambdaTest.

Using Docker Compose

It’s pretty much the same as our previous examples with minor modifications. Here is the docker-compose.yml file we use to integrate our tests with the LambdaTest cloud grid.

# Usage:
#   docker-compose up --force-recreate
version: '2.1'

services:
  #--------------#

  dep:
    image: "elgalu/selenium"
    container_name: selenium
    command: echo 0
    restart: "no" # ensures it does not get recreated

  zalenium:
    image: "dosel/zalenium"
    container_name: zalenium
    hostname: zalenium
    tty: true
    volumes:
      - /tmp/videos:/home/seluser/videos
      - /var/run/docker.sock:/var/run/docker.sock
      - /usr/bin/docker:/usr/bin/docker
    ports:
      - 4445:4444
    depends_on:
      - dep
    command: >
      start --desiredContainers 2
            --maxDockerSeleniumContainers 8
            --screenWidth 800 --screenHeight 600
            --timeZone "Europe/Berlin"
            --videoRecordingEnabled true
            --lambdaTestEnabled true
            --startTunnel false
    environment:
      - HOST_UID
      - HOST_GID
      - LT_USERNAME=${LT_USERNAME}
      - LT_ACCESS_KEY=${LT_ACCESS_KEY}
      - PULL_SELENIUM_IMAGE=true

The configuration is almost similar to previous Travis CI and Docker examples, except that we are configuring the lambdatestEnabled flag to true for the start command. And adding the environment variables for LambdaTest username and access keys using LT_USERNAME and LT_ACCESSKEY. Also, please make sure that the environment variables in the Travis CI project are still intact and available during the build.

dist: trusty
language: java

jdk:
  - oraclejdk8

before_script:
  - curl https://gist.githubusercontent.com/rakesh-vardan/f23ab04fe12512dc6f8ceb20d738f1ad/raw/736edd8a8e7f7276535e253a688e6625cdfd7abb/docker-compose-zel.yml > docker-compose.yml
  - docker-compose up -d --force-recreate
  - docker-compose ps

script:
  - mvn clean install -DsuiteXmlFile=${suiteXmlFile} -Dbrowser=${browser} -DgridHubURL=${gridHubURL}

cache:
  directories:
    - .autoconf
    - $HOME/.m2

The build result would be similar as the services are created by docker-compose, and the test execution happens on the browser containers in LambdaTest.

Which are the most wanted automated testing tools that have climbed the top of the ladder so far? Let’s take a look.

Parallel Testing With Travis CI, Docker And LambdaTest

Till now, we have been discussing running the Selenium UI tests sequentially using multiple options with Travis CI, Docker, and LambdaTest. But as the application enhances we will have more tests in the testing suite. And as the size of the testing suite is big, the time it takes to execute will also be more.

If the execution time is more than we are not meeting the primary goal of test automation, we need to provide early feedback on the product quality to the development team. So how do we achieve this goal without compromising on the testing scope?

The answer is to run the tests in parallel mode wherever feasible to reduce the execution time drastically.

Coming back to our discussion, let us make some changes to the testng.xml file as below to support parallel test execution. Add the required changes in a new file called testng-parallel.xml and add them to our project. We are using the parallel attribute of testNG with a value of tests and configuring the tests to run in parallel with a thread count of 2. As you observe, we are configuring the file to run our tests in Chrome and Firefox browsers from the LambdaTest cloud platform.

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="All Test Suite in parallel" parallel="tests" thread-count="2">
    <test verbose="2" name="travisci-selenium-docker-lambdatest">
        <parameter name="browser" value="GRID_LAMBDATEST_CHROME"/>
        <classes>
            <class name="com.lambdatest.SeleniumTests">
            </class>
        </classes>
    </test>
    <test verbose="2" name="travisci-selenium-docker-lambdatest1">
        <parameter name="browser" value="GRID_LAMBDATEST_FIREFOX"/>
        <classes>
            <class name="com.lambdatest.SeleniumTests">
            </class>
        </classes>
    </test>
</suite>

There are many approaches to running parallel tests using TestNG configuration.

Refer to our detailed blog on configuring TestNG XML and execute parallel testing with detailed explanations.

In order to support the parallel test execution, we need to make the below changes to our .travis.yml file. Here we are using the Zalenium Docker along with LambdaTest to set up our test execution infrastructure.

dist: trusty
language: java

jdk:
  - oraclejdk8

before_script:
  - docker pull elgalu/selenium
  - docker pull dosel/zalenium
  - docker run -d --name zalenium -p 4444:4444 -e LT_USERNAME=${LT_USERNAME} -e LT_ACCESS_KEY=${LT_ACCESS_KEY} -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/videos:/home/seluser/videos --privileged dosel/zalenium start --lambdaTestEnabled true
  - docker ps

script:
  - mvn clean install -DsuiteXmlFile=${suiteXmlFile} -DgridHubURL=${gridHubURL}

cache:
  directories:
    - .autoconf
    - $HOME/.m2

The maven command used to run the tests is as below. We are passing the suite file and the grid URL from the environment variables of Travis CI.

mvn clean install -DsuiteXmlFile=${suiteXmlFile} -DgridHubURL=${gridHubURL}

And the configuration for the job within the Travis CI platform is as below. We are defining the environment variables required for our test execution.

Once these values are set, let’s start our job in Travis CI. After the job is completed successfully, we can see the job log as below. We have added logger statements to our code in the test methods to print the currently used thread id.

Hence from the log, we can see that TestNG creates two threads, and the tests are distributed across them. And a total of 10 tests are executed successfully, five on chrome browser and five on firefox browser in LambdaTest cloud grid.

If we go back to our LambdaTest automation dashboard, we can observe that two sessions are started as the job is executed. One session is for a chrome browser and one for a firefox browser. Also, we can observe that we are using 2 of 5 parallel sessions in the LambdaTest platform — which indicates that our tests are being executed in parallel mode.

Like this, we could play around with configurations available in TestNG, Docker, and LambdaTest and build efficient and robust Selenium test automation suites.

Conclusion

In this post, we have covered some of the important concepts in Travis CI and Docker to build the pipelines for performing the Selenium test automation. We have also discussed integrating our test suites with the LambdaTest cloud grid and leveraging it in our Selenium test automation efforts. and build robust CI/CD pipelines using Travis CI, Docker, and LambdaTest. There are many more exciting features that you can try upon and explore.

Let us know in the comments if you want to have another article on the same topic with advanced capabilities.

Happy Testing and Travis-ci-ing!

Building Automated Testing Pipeline With Semaphore CI And Selenium Grid

RakeshVardhan1 — Wed, 23 Nov 2022 15:33:05 +0000

In today’s digital age, product teams must build products at a brisk pace while maintaining the best quality. Therefore, the methodology to follow and the tool selection to accomplish this can be paramount to get better results. Moreover, software applications don’t just support businesses now; rather, they have become an integral part of a business. Hence, it’s obvious that the product teams deliver a product with speed, reliability, scale, security, quality, and improved collaboration. So companies started following the DevOps model, which is a combination of Development (Dev) and Operations (Ops) teams.

DevOps include the practices such as Continuous Integration and Continuous Delivery (CI/CD), Developing Microservices, Adopting Infrastructure as code, Monitoring and Logging the application, and Communication & collaboration within teams. To rapidly develop a product with speed, teams often use SaaS (Software as a Service) solutions that are available to accomplish the required task without the overhead of setting up or maintaining any infrastructure. One such solution available in the market for CI/CD is Semaphore CI.

In this article, we’ll learn about Semaphore Continuous Integration and dig deeper into using it for building automated testing pipelines with online Selenium Grid.

Let’s begin!

A complete webdriver tutorial that covers what WebDriver is, its features, architecture and best practices.

Overview Of Semaphore CI

Semaphore CI is one of the few CI/CD tools available to accomplish continuous integration and continuous delivery. Other best CI/CD tools in the same space are Jenkins, Azure DevOps, TravisCI, CircleCI, TeamCity Cloud, etc. Semaphore Continuous Integration tool identifies itself as the hosted CI/CD solution for teams that do not have bottlenecks! It is designed and built around increasing developer productivity and reducing the execution time for any pipeline. As per the official website, Semaphore Continuous Integration is the fastest CI/CD solution available in the market, with the ability to deliver the projects in record time.

Refer below to the benchmarking performance of Semaphore CI for a production-grade web application as compared to other CI/CD tools:

From the above survey, it is clear that the Semaphore CI is the winner in terms of shorter execution time, as it has taken the least amount of time to complete operations in the pipeline.

Now you have understood what Semaphore Continuous Integration is, let’s see some of the best features available in Semaphore Continuous Integration.

Features Of Semaphore CI

Below are some of the important features provided by the Semaphore Continuous Integration tool.

Hosted CI solution, no need to manage external servers
Flexibility to build from simple sequential build to multi-stage parallel programmable pipelines
Flexibility to create our environment with native Docker support
Optimize the monorepo workflows
Build, test and deploy on different programming languages and platforms.
Faster builds with smart caching
Support for containerization for pipelines using Docker and Kubernetes.
Ability to debug the failures in seconds by implementing SSH into the running jobs
Support for all deployment strategies — canary, blue-green, etc. including the option to rollback changes.
Built-in Docker registry to store the images.

Before we get into the details of Semaphore CI, let’s brush up on some important concepts around CI/CD. The below video will deep dive into the fundamentals of Continuous Integration, Continuous Deployment, and Continuous Delivery(CI/CD).

However, you can also visit the LambdaTest YouTube channel for videos that will guide you through building an automated testing pipeline using popular CI/CD tools like Jenkins, GitLab, CircleCI, and more.

Mobile Mobile Emulator Online from LambdaTest allows you to seamlessly test your mobile applications, websites, and web apps on mobile browsers and mobile devices.

Getting Started With Semaphore CI

Semaphore Continuous Integration is a commercial CI/CD service offering. It offers three pricing plans for the customers.

Free
Startup
Enterprise Cloud

Though it’s a commercial product, Semaphore CI has a free plan that provides $10 worth of credits (1300 minutes) for running a single job at a time on Linux, macOS, and Docker environments. We will use this option to explore Semaphore Continuous Integration and learn to build pipelines for our test automation examples using cloud Selenium Grid.

The basic prerequisite is that “You need an active GitHub account and a code repository”.

As of now, Semaphore supports creating pipelines for repositories that are hosted on GitHub. However, there is a plan to support repositories in Bitbucket. And supporting other options such as GitLab integration and any git repository is under consideration by the team.

We are using the Demo GitHub repository to discuss examples.

Below are the steps to get started using the Semaphore CI

1- Navigate to Semaphore CI official website and click on the Sign up with GitHub button.

2- Semaphore CI will open up in a new tab. Click on Continue with GitHub button.

3- It takes us to the GitHub authorization page. Authorize the app by clicking on Authorize renderedtext.

4- On the next screen, Semaphore asks you to create an organization. By default, it takes the GitHub username and creates a Semaphore URL for us.

5- Click on Save & Continue.

6- On the next screen, choose the plan for your organization.

7- We can choose either of the options here. Even for the Pro option, we don’t need to enter any credit card details.

8- Once this is completed, we can see our newly configured organization as below.

9- Upon clicking on the organization tile, we could see our workspace as below. We could see all of the work that we have been doing in the organization. Since I have already created some pipelines earlier, we can see all these details. If you are creating it for the first time, you can see an option to set up a new project.

10- Click on the Create New button.

Choose the repository you want to work with. We can either choose the GitHub app or use a GitHub personal token. Let’s select the project and click on Choose.

11- Semaphore initializes the set-up required as below.

12- We can add collaborators required for the project in the Add People or go to the next page.

13- In the Setup workflow, choose the Single Job workflow and click on Looks good, let’s start button.

14- Once done, Semaphore Continuous Integration starts the pipeline and executes the jobs inside the pipeline.

That’s it! This is the entire process of setting up a new account with Semaphore Continuous Integration and creating a new sample pipeline.

In the next section, let’s understand some important concepts and terminology used in Semaphore Continuous Integration and then build new workflows and pipelines.

In this article, we look at what Regression testing is, its importance and types, and how to perform it.

Important Concepts In Semaphore CI

Similar to other CI/CD solutions, Semaphore Continuous Integration has its own set of terminology while working with pipelines:

Workflow: Workflow in Semaphore CI is similar to a Jenkins Pipeline. Semaphore uses pipelines, blocks, tasks, and promotions in the workflow to manage the application CI process. Each workflow for a project can have multiple pipelines.
Pipeline: A pipeline is a collection of similar blocks within the same workflow.
Block: Blocks are the building components of a pipeline. Each block in Semaphore CI consists of an optional name and task as a mandatory value. Blocks define steps in the pipeline on what to do as part of the pipeline. By default, all the blocks in the pipeline run sequentially by Semaphore CI. However, we can also define blocks to run in parallel mode by using block dependencies.
Task (Mandatory): Task specifies what to do in the block. The task consists of jobs, epilogue, prologue, and secrets.
Job: Jobs in the Semaphore specify the commands to execute in a pipeline. Each job is run in an isolated clean virtual environment created by the Semaphore. Examples include checking out the code, running tests using Maven, etc.
Promotion: Promotions in Semaphore are the junction blocks used for deployment purposes and promoting builds to different environments.
Secret: Secrets in Semaphore are used to store sensitive information for a workflow such as API keys, passwords, etc., which are not committed to the version control system.
Agent: Virtual environment to run the workflows is created by Semaphore, post which it is ready for us to use. These machines are called agents. They perform the actual execution of the job. Each agent consists of a machine type and OS image used to build the agent. Semaphore supports agents on Linux and macOS environments.

The below picture from the official Semaphore documentation gives a high-level overview of these concepts.

Test your website or web app online for iOS browser compatibility. Perform seamless cross browser testing on the latest iPhone tester. Try for free!

Building A Pipeline With Semaphore CI

If you have noticed in the getting started section, Semaphore has created a new branch called setup-semaphore and added a configuration file semaphore.yml in the .semaphore directory in our project. Basically, the idea here is to create a pull request to the master/main branch from the setup-semaphore branch once all the pipeline steps are defined.

Based on the pull request trigger, which is configured by default, Semaphore starts the workflow and executes the jobs within the pipeline.

We have created the pull request to master and we will be working in the master branch only for all our examples.

There are two options available to create a new workflow in Semaphore.

Using the Visual Builder
Using the declarative YAML configuration in semaphore.yml

Let us first start configuring our workflow using the Visual Builder. But, before that, let’s try to understand the test scenario that we are trying to configure in Semaphore Continuous Integration.

Open a specified browser.
Open the GitHub page and verify the title.
Open the Gmail page and verify the title.
Open the LambdaTest page and verify the title.

We have set up our Selenium automation project and TestNG to run the tests in any browser. We use the BrowserFactory and BaseTest to instantiate the browser based on the input and pass it on to our tests. Please refer to the Demo GitHub repository for complete project code.

We are using TestNG and adding appropriate annotations for our tests. We have also configured the testng.xml file to include what tests to run.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="All Test Suite">
    <test verbose="2" name="firefox tests">
        <parameter name="browser" value="LOCAL_FIREFOX"/>
        <parameter name="gridHubURL" value="http://localhost:4444/wd/hub"/>
        <classes>
            <class name="org.lambdatest.FirstTest"/>
            <class name="org.lambdatest.SecondTest"/>
        </classes>
    </test>
</suite>

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="All Test Suite">
    <test verbose="2" name="firefox tests">
        <parameter name="browser" value="LOCAL_FIREFOX"/>
        <parameter name="gridHubURL" value="http://localhost:4444/wd/hub"/>
        <classes>
            <class name="org.lambdatest.FirstTest"/>
            <class name="org.lambdatest.SecondTest"/>
        </classes>
    </test>
</suite>

Finally, we will use the below maven command to trigger the tests.

mvn clean install -DsuiteXmlFile=testng.xml

Complete code for this project is available here on the GitHub repository.

Adding A Pipeline With Visual Builder

With all the prerequisites in place, let’s start with our first pipeline using the Visual Builder in the Semaphore CI.

Go to the organization URL, and open the project that we have imported from GitHub. Select the branch as master and click on the edit workflow button.

We can see the values below for the pipeline section.

As we see here, Semaphore has chosen default values for The Agent, OS Image, and the Machine types. If needed, we can change these values and proceed with creating the Job. If any other steps need to be performed before and after running the pipeline, we can configure the commands in the Prologue and Epilogue sections, respectively.

For now, let’s take all the default values and change the name to First Pipeline in Semaphore and click on the Job.

Add a name for the block, give the specified commands in the jobs section and keep other values at their defaults. Here the command checkout is mandatory to clone the repository and create the workspace. The next one is the maven command that we need to run our UI tests in Selenium.

Click on Run the workflow. Semaphore asks to commit the changes to the code repository. Add a comment and click on Looks good, let’s go. Semaphore initiates the workflow and starts executing the job that is configured for our project. As we observe in the below screen, the workflow is in Running status.

Once the workflow is completed, we could check the logs as below. Next, the Semaphore starts creating the fresh virtual environment for our job, loads all the required variables, if any, checks out the code from the repository, and runs the build commands specified.

From the below screen, we could observe that our pipeline execution is successful, and all our tests are passed!

Here is the summary of the tests that are executed as part of our job.

Finally, the status is updated as Passed at the workflow level.

A complete Webdriver tutorial that covers what WebDriver is, its features, architecture and best practices.

Configuring A Pipeline With semaphore.yml

In our previous run, we have leveraged the Visual Builder option available in Semaphore CI to add the configuration for the pipeline. However, you can also add the configuration using a declarative YAML file — semaphore.yml

You have to click on the Edit workflow button and then click on the semaphore.yml file to open the configuration file on the web. We can see the below configuration is automatically available for us. It is because of the configuration we did using the visual builder option in the previous step. Therefore, the same instructions are replicated here.

Let’s look at the configuration in detail.

version — version refers to the pipeline YAML version used in the configuration
name — the name of the pipeline
agent — agent defines the virtual environment, i.e., software and hardware that needs to be created to run our job
machine — machine property is added to the agent only, and it has two properties — type and os_image
type — virtualized hardware that is available for us to use in jobs
os_image — operating system to be used for the jobs
blocks — blocks define steps in the pipeline on what to do as part of the pipeline
task — the task is a mandatory property in blocks which refers to the steps needed inside a block
env_vars — key-value pair that holds the environment variables as part of a job
jobs — property where we mention the actual commands to be executed along with a name to it
commands — an array of strings that holds the commands to execute for a job

Now that we have understood the semaphore.yml configuration let’s discuss the next example. We use the below semaphore.yml file for our next pipeline execution and see if we could run the pipeline in Semaphore Continuous Integration. As you observe, we are running our tests in the Chrome browser this time.

Semaphore.yml file

version: v1.0
name: First Pipeline with semaphore
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Tests in Chrome
    task:
      prologue:
        commands:
          - checkout
      jobs:
        - name: Run tests
          commands:
            - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=LOCAL_CHROME
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml

Some of the new properties are mentioned below:

prologue — This is executed before the commands of each job of a task item. In our example, we have added the checkout command to the prologue block to check the source code before our test execution.
epilogue — This is executed after the commands of each job item of a task. Semaphore has a beta feature at the moment to publish the test results from the pipeline execution without any additional configuration. However, since we are using TestNG in our example, we get the XML reports post-execution. Hence we configured the XML files as part of the epilogue block to check the execution results in a more readable way.

Also, we are passing the browser property to the test suite using the below maven command.

mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=LOCAL_CHROME

Once the pipeline is executed with the new configuration specified in the semaphore.yml file, we can see the below log. Our execution is successful even in the Chrome browser also!

From the logs, we could see that the Chrome browser instance is launched.

And the tests are executed successfully.

The workflow view is shown below:

Please take a look at the expanded logs for the test results publishing command that we specified in the epilogue section. In addition, Semaphore is publishing the JUnit XML results to the summary page.

Navigate to the Summary tab, and we can see the detailed test execution results there.

If there is a failure in our tests, the view displays below — with the details to easily debug the issue.

Perform browser Automation testing on the most powerful cloud infrastructure. Leverage LambdaTest automation testing for faster, reliable and scalable experience on cloud.

Building A Complex Workflow With semaphore.yml

With the knowledge that we have acquired till now, we could run the pipelines in the Semaphore Continuous Integration tool. We have discussed using Visual Builder and semaphore.yml to create and run a simple UI test pipeline. How about taking our experience to the next level and creating a complex workflow for our project? Let’s see how we can do it.

Add the below code in the semaphore.yml file and run the workflow.

Semaphore.yml file for complex flow — sequential

version: v1.0
name: Complex Pipeline with multiple browsers - sequential
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Build
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      jobs:
        - name: Build
          commands:
            - checkout
            - mvn -q package -DskipTests=true
            - cache store
  - name: API Smoke tests
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run Smoke API tests
          commands:
            - mvn clean install -DsuiteXmlFile=testng-api.xml
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml
  - name: Firefox tests
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run tests in Firefox
          commands:
            - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=LOCAL_FIREFOX
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml
  - name: Chrome tests
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run tests in Chrome
          commands:
            - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=LOCAL_CHROME
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml

From the workflow overview page, we could see the below execution summary. First, the project is built, then the API smoke tests are executed, then the UI tests are executed in Firefox browser followed by UI test execution in Chrome browser.

We are using the env_vars property for storing the maven local repository path. The env_vars property is a key-value pair with the name and value for the variable. Next, we are using the Semaphore CI caching feature to cache and reuse the project dependencies.

We have used the cache store command to store the cache and cache restore to restore the saved cache. As you have noted, along with our previous UI tests using Selenium, we have added a simple API test as part of our
suite using REST Assured library. And we configured to run the API tests using a separate TestNG XML file — testng-api.xml

Since we haven’t declared any dependencies between the jobs inside the block, the jobs are executed in sequential order by default one after the other. So let’s tweak our semaphore.yml as below and see if we can run the same in parallel mode.

Add the below code in the semaphore.yml file and run the workflow.

Semaphore.yml file for complex flow — Parallel

version: v1.0
name: Complex Pipeline with multiple browsers - parallel
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Build
    dependencies: []
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      jobs:
        - name: Build
          commands:
            - checkout
            - mvn -q package -DskipTests=true
            - cache store
  - name: API Smoke tests
    dependencies:
      - Build
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run Smoke API tests
          commands:
            - mvn clean install -DsuiteXmlFile=testng-api.xml
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml
  - name: Firefox tests
    dependencies:
      - API Smoke tests
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run tests in Firefox
          commands:
            - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=LOCAL_FIREFOX
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml
  - name: Chrome tests
    dependencies:
      - API Smoke tests
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run tests in Chrome
          commands:
            - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=LOCAL_CHROME
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml

From the workflow overview page, we could see the above test execution summary. First, the project is built, then the API smoke tests are executed. As per the semaphore.yml configuration, we have added API Smoke tests as the dependency for jobs to run Firefox tests and Chrome tests.

As soon as the API Smoke tests job is completed, Semaphore triggers the other jobs simultaneously (i.e. the UI tests are executed in Firefox browser and Chrome browser simultaneously). This small configuration change enables the execution of the builds in less time and improves the overall efficiency of the pipelines. Therefore, we could achieve parallel execution without any code changes on our project, along with zero changes in testng.xml.

Semaphore CI Pipeline With Docker

Till now, we have been running our tests on the browsers launched in the virtual infrastructure created by the Semaphore CI. Semaphore Continuous Integration also has native support to create and run the pipelines using docker. Docker CLI is already installed in all Semaphore virtual machines; hence we can use it without additional configuration. Let’s modify our configuration to run the same pipeline in browser containers launched by docker. Docker is one of the most widely used containerization technologies. To learn more about Docker and execute UI Selenium tests using Docker, please refer to our detailed article on how to run Selenium tests in Docker.

Add the below code in the semaphore.yml file and run the workflow in docker.

Semaphore.yml file for complex flow — Docker Containers

version: v1.0
name: Complex Pipeline with multiple browsers - docker
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Build
    dependencies: []
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      jobs:
        - name: Build
          commands:
            - checkout
            - mvn -q package -DskipTests=true
            - cache store
  - name: API Smoke tests
    dependencies:
      - Build
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run Smoke API tests
          commands:
            - mvn clean install -DsuiteXmlFile=testng-api.xml
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml
  - name: Firefox tests
    dependencies:
      - API Smoke tests
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run tests in Firefox
          commands:
            - docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-firefox:4.0.0-rc-1-prerelease-20210618
            - docker ps
            - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_FIREFOX -DgridHubURL=http://localhost:4444/wd/hub
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml
  - name: Chrome tests
    dependencies:
      - API Smoke tests
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run tests in Chrome
          commands:
            - docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:4.0.0-rc-1-prerelease-20210618
            - docker ps
            - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_CHROME -DgridHubURL=http://localhost:4444/wd/hub
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml

As you take a look at the semaphore.yml configuration, there are some minor changes that we have made to enable the execution in docker containers. We will use the below docker run commands to spin specific browser containers on port 4444 and point our tests to run on the relevant grid hub URL with maven arguments.

docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-firefox:4.0.0-rc-1-prerelease-20210618

mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_FIREFOX -DgridHubURL=http://localhost:4444/wd/hub

docker run -d -p 4444:4444 -v /dev/shm:/dev/shm selenium/standalone-chrome:4.0.0-rc-1-prerelease-20210618

mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_CHROME -DgridHubURL=http://localhost:4444/wd/hub

Once the execution is completed, we can see the workflow overview as below.

If we take a look at each of the jobs for our UI tests that are using docker containers to execute the tests, we can see the below log information.

Docker starts the stand-alone firefox container on port 4444 as we see in the output of the docker ps command. Then, using the gridHubURL we have sent the test requests to the container on the same port. Hence the tests are executed successfully on Firefox containers and the result is passed.

Similarly, the Chrome container is started on port 4444, tests are executed and the execution is passed.

Semaphore CI Pipeline With docker-compose

We have seen multiple examples of building test automation pipelines till now. We would be able to use Docker for spinning up the containers and executing our tests in them. Let us take this forward and explore building the automation pipelines with docker-compose in Semaphore CI.

Docker-compose is a utility provided by Docker for defining and running the multi-container docker applications. We have specified all the instructions required to start the services in a declarative YAML file and trigger it with a simple docker compose up command.

Let’s modify our configuration to run the same pipeline in browser containers launched by the docker-compose. First, we need to add the below code in the semaphore.yml file and run the workflow.

Semaphore.yml file for complex flow — Docker-Compose

# To execute this docker-compose yml file use `docker-compose -f docker-compose-v3.yml up`
# Add the `-d` flag at the end for detached execution
# To stop the execution, hit Ctrl+C, and then `docker-compose -f docker-compose-v3.yml down`
version: "3"
services:
  chrome:
    image: selenium/node-chrome:4.0.0-rc-1-prerelease-20210618
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6900:5900"

  edge:
    image: selenium/node-edge:4.0.0-rc-1-prerelease-20210618
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6901:5900"

  firefox:
    image: selenium/node-firefox:4.0.0-rc-1-prerelease-20210618
    volumes:
      - /dev/shm:/dev/shm
    depends_on:
      - selenium-hub
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
    ports:
      - "6902:5900"

  selenium-hub:
    image: selenium/hub:4.0.0-rc-1-prerelease-20210618
    container_name: selenium-hub
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"

Once the execution is completed, we could see the below details on the workflow overview page.

If you have noticed the semaphore.yml configuration file, there is a change in the way we set up the browser containers. We are using the below commands with docker compose and starting the required services.

$ curl https://gist.GitHubusercontent.com/rakesh-vardan/c1dcf6531b826fad91f18c285d566a71/raw/ad90a18fe5e70f2d6aea06621b76d6e0329a4aab/docker-compose-sel.yml > docker-compose.yaml
$ docker-compose up -d
$ docker-compose ps

We have added the services in the docker-compose.yml file and downloaded the content for our execution. Once it is available in the build path, we have invoked the docker-compose up command to start the services using docker.

From the log, we could observe that there are 4 containers created for Chrome, Edge, and Firefox browsers and Selenium hub. Since the hub is running on port 4444, we have redirected our test requests to the same hub URL.

As per the below screen, the tests have been executed successfully.

There is one difference in the configuration in this example. If you have observed, we have added the docker-compose commands in the prologue section of the task only, and instead of creating a separate block for each UI test, we have added multiple jobs in the same block itself. This is because all jobs in Semaphore run in an isolated environment in virtual machines. So the services that start in one job are not automatically available to other jobs in the workflow. So to share the services across, we have added the commands in the prologue section.

Building Semaphore CI Pipeline With LambdaTest

We are good with our previous examples and can execute the tests in Semaphore pipelines. However, suppose we have requirements for viewing the execution live, checking the execution logs, parallel execution, and recording execution sessions. In that case, this functionality is not readily available for us to include in our example. So, either we need to change the code or set up additional tools to enable this feature for our pipelines. So do we have a solution here? Yes, we can use a cloud Selenium Grid like LambdaTest.

LambdaTest is a cross browser testing cloud platform to perform automation testing on 2000+ combinations of real browsers and operating systems. It is very easy to configure a project using LambdaTest and start leveraging the benefits out of it. So, let’s integrate our pipeline with the LambdaTest cloud grid solution and see how it’s working.

To start using the LambdaTest cloud, navigate to the LambdaTest Registration Page and sign-up for a new account (which is free to start with!). Once the LambdaTest account is activated, go to the LambdaTest Profile Section and note the username and access key. We would need this LambdaTest username and accesskey to execute our tests on the LambdaTest platform.

Let’s modify our configuration to run the same pipeline in browsers launched by the LambdaTest cloud grid. Add the below code in the semaphore.yml file and run the workflow.

Semaphore.yml file for complex flow — LambdaTest

version: v1.0
name: Complex Pipeline with multiple browsers - parallel with LambdaTest
agent:
  machine:
    type: e1-standard-2
    os_image: ubuntu1804
blocks:
  - name: Build
    dependencies: []
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      jobs:
        - name: Build
          commands:
            - checkout
            - mvn -q package -DskipTests=true
            - cache store
  - name: API Smoke tests
    dependencies:
      - Build
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run Smoke API tests
          commands:
            - mvn clean install -DsuiteXmlFile=testng-api.xml
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml
  - name: Firefox tests
    dependencies:
      - API Smoke tests
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run tests in Firefox
          commands:
            - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_LAMBDATEST_FIREFOX
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml
      secrets:
        - name: LT
  - name: Chrome tests
    dependencies:
      - API Smoke tests
    task:
      env_vars:
        - name: MAVEN_OPTS
          value: '-Dmaven.repo.local=.m2'
      prologue:
        commands:
          - checkout
          - cache restore
      jobs:
        - name: Run tests in Chrome
          commands:
            - mvn clean install -DsuiteXmlFile=testng.xml -Dbrowser=GRID_LAMBDATEST_CHROME
      epilogue:
        always:
          commands:
            - test-results publish target/surefire-reports/junitreports/*.xml
      secrets:
        - name: LT

Before triggering the build, we need to make sure that the below environment variables are set as secrets in the project settings in Semaphore CI. Secrets in Semaphore Continuous Integration are the objects that have global scope within an organization. These secrets can be accessed by the jobs or pipelines to which they have been connected.

LT_USERNAME - Username from the LambdaTest profile page.
LT_ACCESS_KEY - Access key obtained from the LambdaTest profile page.

The grid hub URL is formed based on the values that we set in the environment variables, as shown below:

https://<username>:<accessKey>@hub.lambdatest.com/wd/hub

Where username is the Username, accessKey is the access key from the LambdaTest account.

Once the execution is completed, we could see the below details on the workflow overview page in the Semaphore CI.

And from the detailed logs, we could see that the execution is successful in Firefox and Chrome browsers. Furthermore, we can also see the command used to execute the tests.

This looks good, but how about the execution statistics and information. Once the Semaphore CI completes the test execution, we can navigate to the LambdaTest Automation Logs to see our test execution in the Timeline view. To navigate to the Automation logs view, click on the test case to see other details such as execution recording, logs, etc.

As we observe, we can see that 2 out of 5 parallel sessions are running in LambdaTest — one is on Chrome browser, and another is on Firefox browser.

We could observe that 6 tests are executed in browsers available on the LambdaTest cloud grid from the timeline view.

Here is the detailed view of automation logs, along with a recording of the test execution. Of course, we could also build very useful analytics like coverage, pass-failed ratio, and minutes consumed, etc., from the test execution results and build ready-to-use metrics.

Isn’t it interesting? Without any code changes on our part, we can easily integrate our automation suite with LambdaTest’s cloud grid solution and benefit from the features that it provides to us.

Semaphore CI Additional Capabilities

Let’s throw some light into some of the additional capabilities provided by the Semaphore CI. First, we will discuss the command-line options and the REST API provided by the Semaphore Continuous Integration tool.

Command Line Interface(CLI)

Semaphore command line interface(CLI) is called sem. It enables us to create and manage the Semaphore projects and resources directly from the command line terminal. Sem is a very powerful CLI utility that comes with the power of managing all the resources of Semaphore — workflows, pipelines, jobs, blocks, organizations, etc.

To get started using the sem cli, open a terminal either in a Linux machine or a macOS machine. Currently, the sem cli is not available to Windows OS. Issue the below command to download and run the script to have the cli available on your machine.

curl https://storage.googleapis.com/sem-cli-releases/get.sh | bash

Once done, use the sem connect command along with the organization token to connect to the organization.

sem connect ORGANIZATION.semaphoreci.com ACCESS_TOKEN

Get your access token from the below page in Semaphore CI. If there is no token available for your account, create one for yourself using the Generate API token button.

Once you are connected to the required organization, we can issue the sem commands to manage our resources in Semaphore CI. Check the available option using the sem –help command.

Let us see if we can get all the pipelines available in our project. So issue the sem command as below along with the project name. So we could see all the pipelines that we have used till now along with their details.

sem get pipelines -p semaphoreci-selenium-lambdatest

As we see below, we can check all the logs generated as part of the specific job.

Most of the actions we perform on the Semaphore web interface can be performed using the command line option sem. Please refer to the official Semaphore CLI documentation to get other available options in sem CLI.

Conclusion

Thank you for reading our detailed article on building test automation pipelines using Semaphore CI. We discussed different approaches for creating the workflows in Semaphore, starting with a simple one. Then we discussed the complex pipelines of both sequential and parallel execution. We have also discussed leveraging docker in our pipelines, including the docker-compose utility. Finally, we discussed integrating the workflows with the cloud grid solution — LambdaTest. As part of our examples, we have covered some of the essential concepts from Semaphore CI like workflows, blocks, jobs, pipelines, and secrets, etc. We also touched upon using the Semaphore Continuous Integration using the CLI and REST API. We hope this article can help you get started with building pipelines with Semaphore CI.

Happy Testing!