DEV Community: Abubakkar Sithik

Running Postman Collections in Jenkins

Abubakkar Sithik — Sun, 09 Feb 2025 11:32:36 +0000

As APIs become increasingly crucial in modern software development, maintaining comprehensive test coverage is essential. In this post, I'll share a complete solution for running Postman collections using Jenkins, perfect for regression testing and continuous integration.

The Problem

Many organizations maintain extensive API test suites using Postman collections. However, running these tests manually is time-consuming and prone to human error. We needed a solution that would:

Automate the execution of multiple Postman collections
Generate detailed HTML reports
Send email notifications with results
Support different environments (QA/Staging/Prod)
Integrate seamlessly with our CI/CD pipeline

The Solution

I've created a Jenkins pipeline that automates the entire process using Newman (Postman's command-line collection runner). Here are the key features:

Parallel execution of multiple Postman collections
Customizable HTML reports with detailed test results
Email notifications with a summary table
Environment-specific configurations
Artifact archiving for historical tracking

Prerequisites

Jenkins server
NodeJS installed on Jenkins
Postman collections and environments
Postman API key

Implementation Details

1. Pipeline Configuration

The pipeline uses Jenkins parameters to make it flexible:

parameters {
    text(
        name: 'COLLECTIONS_JSON',
        defaultValue: '''[
            {"name": "Sample_Collection",     
             "url": "https://api.postman.com/collections/YOUR_COLLECTION_ID?apikey=${POSTMAN_API_KEY}"
                }
            // ... other collections
        ]''',
        description: 'JSON array of collections to run'
    )
    choice(
        name: 'ENVIRONMENT',
        choices: ['QA', 'Prod'],
        description: 'Select the environment'
    )
}
// ... The complete code is available on GitHub (https://github.com/abu-sithik/jenkins_postman_pipeline).

2. Newman Integration

The pipeline installs Newman and its HTML reporter:

stage('Install Newman') {
    steps {
        sh 'npm install -g newman'
        sh 'npm install -g newman-reporter-htmlextra'
    }
}

3. Collection Execution

Each collection runs with environment-specific configurations:

def runCollection(collection, environment_url) {
    // ... collection execution logic
    // ... The complete code is available on GitHub (https://github.com/abu-sithik/jenkins_postman_pipeline).
}

4. HTML Report Generation

The solution includes a custom HTML report generator:

def generateHtmlReport(reportData) {
    // ... HTML generation logic
    // ... The complete code is available on GitHub (https://github.com/abu-sithik/jenkins_postman_pipeline).
}

5. Email Notifications

Results are automatically emailed to stakeholders:

def sendEmail(htmlContent, emailConfig) {
    emailext(
        subject: "${emailConfig.subject}",
        body: htmlContent,
        to: emailConfig.recipients,
        from: emailConfig.sender,
        mimeType: 'text/html'
    )
}

Setting Up the Pipeline

1. Required Plugins

Install the following Jenkins plugins:

NodeJS Plugin
Email Extension Plugin
Pipeline Plugin
Credentials Plugin

Manage Jenkins > Manage Plugins > Available > Search and install each plugin

2. NodeJS Configuration

Navigate to Manage Jenkins > Tools
Click "Add NodeJS"
Configure as follows:
- Name: NodeJS_22
- Install automatically: Check
- Version: Select latest LTS version
Click Save

3. Credentials Setup

Navigate to Manage Jenkins > Manage Credentials
Click on "Jenkins" under Stores scoped to Jenkins
Click "Global credentials"
Click "Add Credentials"
Configure:
- Kind: Secret text
- Scope: Global
- Secret: Your Postman API key
- ID: POSTMAN_API_KEY
- Description: Postman API Key for Collection Runner

4. Email Configuration

Navigate to Manage Jenkins > Configure System
Find "Extended E-mail Notification"
Configure:
- SMTP server
- SMTP port
- Credentials if required
- Default recipients
- Default subject
- Default content

Pipeline Configuration

1. Create Pipeline Job

Click "New Item" on Jenkins dashboard
Enter name for your pipeline
Select "Pipeline"
Click OK

2. Configure Pipeline

In pipeline configuration:
- Select "Pipeline script" or "Pipeline script from SCM"
- If using SCM:
  - Select Git
  - Enter repository URL
  - Specify branch
  - Script Path: Jenkinsfile

3. Environment Configuration

Update the ENVIRONMENT_URLS variable in Jenkinsfile:

ENVIRONMENT_URLS = [
    QA: [id: 'YOUR_QA_ENV_ID', url: ''],
    Staging: [id: 'YOUR_STAGING_ENV_ID', url: ''],
    Production: [id: 'YOUR_PROD_ENV_ID', url: '']
]

Postman Setup

1. Collection Preparation

Create/organize your Postman collections
Ensure all environment variables are properly set
Add appropriate tests to requests
Get collection IDs from Postman

2. Environment Setup

Create environments in Postman for each target (QA, Staging, Production)
Set appropriate variables
Get environment IDs from Postman

First Run

Open pipeline in Jenkins
Click "Build with Parameters"
Enter test configuration:

   [
       {
           "name": "Test_Collection",
           "url": "https://api.postman.com/collections/YOUR_COLLECTION_ID?apikey=${POSTMAN_API_KEY}"
       }
   ]

Select environment
Enter email recipients
Click Build

Conclusion

This solution has significantly improved our API testing process by:

Reducing manual effort
Providing consistent test execution
Generating comprehensive reports
Enabling early detection of API issues

The complete script is available on GitHub.

Cleaning Up Stale Branches to Speed Up CI/CD and Reduce GitHub API Load

Abubakkar Sithik — Wed, 05 Feb 2025 10:52:49 +0000

The Problem: Too Many Stale Branches

In my company, we manage multiple microservices, each with its own repository. Over time, our repositories accumulated hundreds—sometimes ~800+ stale branches. Many of these were created over the span of 6+ years and were never deleted. This led to several issues:

Jenkins build auto-start times skyrocketed to ~20 minutes due to multi-branch scanning.
GitHub API request limits became a concern as our build processes constantly queried branch metadata.
Codebase clutter made it difficult to navigate active branches.

We needed a safe and automated way to identify and delete these branches without disrupting active development. Enter GitHub CLI (gh) + Bash!

The Solution: Automating Stale Branch Cleanup

To solve this problem, I wrote a Bash script leveraging the GitHub CLI to:

Fetch all branches in a repository.
Check their last commit date.
Log stale branches to a CSV file (for review).
(Optional) Automatically delete branches older than a specified date.

The Script:

#!/bin/bash

# Ensure gh CLI is installed and authenticated
if ! command -v gh &> /dev/null; then
    echo "GitHub CLI (gh) is not installed. Please install it first."
    exit 1
fi

# Get repository name (default: current repo)
if [ -z "$1" ]; then
    REPO=$(gh repo view --json nameWithOwner -q .nameWithOwner)
else
    REPO=$1
fi

OUTPUT_FILE="stale_branches.csv"

echo "branch_name,last_commit_date" > $OUTPUT_FILE

echo "Fetching branches for $REPO..."
branches=$(gh api repos/$REPO/branches --paginate --jq '.[].name')

target_date="2024-01-01T00:00:00Z"
count=0
processed=0

echo "Analyzing branches..."
for branch in $branches; do
    ((processed++))
    echo -ne "Progress: $processed branches processed\r"

    last_commit_date=$(gh api repos/$REPO/commits/$branch --jq '.commit.committer.date' 2>/dev/null || echo "")
    if [ ! -z "$last_commit_date" ] && [ "$last_commit_date" \< "$target_date" ]; then
        echo "$branch,$last_commit_date" >> $OUTPUT_FILE
        ((count++))
    fi
done

echo -e "\nFound $count stale branches (last commit before $target_date)"
echo "Results written to $OUTPUT_FILE"

# Deleting stale branches
if [ $count -gt 0 ]; then
    echo "Deleting stale branches..."
    deleted=0
    failed=0
    while IFS=, read -r branch_name last_commit; do
        if [ "$branch_name" != "branch_name" ]; then  # Skip header
            if gh api -X DELETE repos/$REPO/git/refs/heads/$branch_name >/dev/null 2>&1; then
                ((deleted++))
                echo "Deleted: $branch_name"
            else
                ((failed++))
                echo "Failed to delete: $branch_name"
            fi
        fi
    done < "$OUTPUT_FILE"
    echo "Successfully deleted: $deleted branches"
    [ $failed -gt 0 ] && echo "Failed to delete: $failed branches"
fi

echo "Total branches processed: $processed"

How It Works:

Fetches all branches in the given repository.
Checks the last commit date for each branch.
Logs stale branches (last commit before 2024-01-01) to a CSV file.
Deletes stale branches automatically (if enabled).

By running this script, we cleaned up hundreds of old branches, reducing our Jenkins build start time significantly and staying within GitHub’s API limits.

Lessons Learned & Best Practices

1️⃣ Automate but Review First

Before enabling automatic deletion, review the CSV file. This prevents accidental deletion of branches that might still be needed.

2️⃣ Protect Important Branches

Make sure to exclude main, develop, or any critical branches if necessary.

3️⃣ Schedule Regular Cleanup

Stale branches will accumulate again. Automate this script via a scheduled GitHub Action or Jenkins job to run quarterly or semi-annually.

4️⃣ Use GitHub API Efficiently

To avoid hitting GitHub API rate limits, paginate API calls and consider caching results if working with many repositories.

Conclusion

This simple script helped us streamline our GitHub repositories, improve CI/CD performance, and prevent API overuse. If your team struggles with stale branches, give this a try!

👉 What are your strategies for handling stale branches? Let me know in the comments!

Automating Mirroring Between Bitbucket and GitHub

Abubakkar Sithik — Mon, 20 May 2024 10:55:53 +0000

💡 The primary objective of this process is to demonstrate the feasibility and functionality of setting up automated mirroring between Bitbucket and GitHub repositories.

Overview

This article outlines the steps to mirror a Bitbucket repository to a GitHub repository. The process involves configuring Bitbucket Pipelines to automate the synchronization of code changes between the two repositories. This setup can be particularly useful for teams looking to leverage GitHub’s enhanced collaboration features and seamless integration with existing toolsets while still maintaining their existing Bitbucket repositories.

Steps to Automate Mirroring

The following steps will guide you through establishing automated mirroring from Bitbucket to GitHub:

1. GitHub Repository Setup

Create a new repository on GitHub. For this example, we'll refer to it as "github-repo".

2. Bitbucket Pipelines Configuration

Enable Pipelines under Repository settings > Pipelines > Settings on Bitbucket.

3. SSH Key Generation (Bitbucket)

Generate SSH keys under Repository settings > Pipelines > SSH keys on Bitbucket.
Copy the public key to the clipboard.

4. Known Hosts Configuration (Bitbucket)

Add github.com as the Host address under Known hosts.
Click Fetch followed by Add host.

5. Deploy Key Configuration (GitHub)

Add the public key under Settings > Security > Deploy keys > Add deploy key on GitHub.
Enable write access for the deploy key.

6. Access Key Configuration (Bitbucket)

Add the public key under Repository settings > Security > Access keys > Add key on Bitbucket.

7. Pipeline Configuration (Bitbucket)

Create a bitbucket-pipelines.yml file inside the Bitbucket repository with the following content:

image: atlassian/default-image:3

pipelines:
  # Automated pipeline for syncing Bitbucket repository with its GitHub mirror, ensuring real-time updates across platforms
  branches:
    '*':
      - step:
          name: Sync GitHub Mirror
          image: alpine/git:latest
          clone:
            enabled: false
          script:
            - git clone --bare git@bitbucket.org:your-org/bitbucket-repo.git
            - cd bitbucket-repo.git
            - git push --mirror git@github.com:your-org/github-repo.git

This configuration ensures that any code push to the Bitbucket repository will initiate a Bitbucket pipeline job, triggering automatic mirroring to the GitHub repository. The synchronization of code changes prepares the team for a seamless transition to using GitHub as their primary repository platform.

Conclusion

By following these steps, you can automate the mirroring of a Bitbucket repository to a GitHub repository. This setup ensures that any code push to the Bitbucket repository triggers an automatic pipeline job, mirroring the changes to the GitHub repository. This process is beneficial for teams looking to utilize GitHub’s advanced features while maintaining their existing workflows on Bitbucket.

Note: Further optimization and fine-tuning may be required for production deployment.

How to Build a Selenium UI Testing Framework

Abubakkar Sithik — Mon, 12 Jun 2023 13:26:47 +0000

Selenium UI Testing Automation Framework

I recently created a Selenium UI testing automation framework for a sample flight/hotel booking website. The framework is based on the PageFactory and Fluent Interface patterns, and it uses Selenium, TestNG, and Extent Reports.

The framework is designed to be easy to use and maintain. It includes a set of page classes that represent the different pages of the website, as well as a set of test classes that exercise the page classes. The test classes are written using the Fluent Interface pattern, which makes them more readable and maintainable.

The framework also includes a report generator that creates interactive and detailed reports for the test results. The reports include information such as the test name, the test status, and the test steps.

This framework can be used for UI automation testing of any web application. I am confident that it will be a valuable tool for other developers as well.

Framework Details

Test Application

For this framework, I used ClearTrip as the sample flight/hotel booking website.

Framework - What and Why?

A framework is a pre-existing codebase that provides common functionalities and solves basic tasks for software requirements. By building on top of a framework, developers can focus on solving specific requirements without reinventing the wheel. The chosen framework serves as the foundation for implementing the software requirements specific to the project.

Project Structure

The project follows a standard directory structure. The page classes and related files are located in the src/main/java/com/cleartrip/casestudy path, while the test classes and related files can be found in the src/test/java/com/cleartrip/casestudy path. This separation allows for better organization and maintainability.

Key features of the framework:

Page classes for each page under test, utilizing the PageFactory model to find and initialize web elements.
Test classes that create instances of the page classes to access their methods, decoupling tests from the page elements.
Static instantiation of the WebDriver to ensure only one instance is shared across all test classes.
Relevant dependencies and plugins defined in the pom.xml file.

Packages

Main Package

The src/main/java/ package contains the core framework code. All the page classes and related files are located in this main package. Detailed documentation can be found in the Javadoc at <working-dir>/doc/index.html.

Test Package

The src/test/java/ package contains the actual test classes. The subpackage com.cleartrip.casestudy.tests holds all the TestNG test classes related to the ClearTrip application. The Javadoc for these test classes can be found at <working-dir>/doc/com/cleartrip/casestudy/tests/package-summary.html.

Reports

With the Extent library, we can generate interactive and detailed reports for our UI test results. The test reports can be found at workingDir/ExtentReports/ExtentReportResults.html.

Installation

To set up and run the UI automation framework on your local system, follow these steps:

Steps to follow to set up Selenium in the local system:

1. Install Java:

Check if Java is already installed by running java -version and javac -version in the terminal.
If Java is not installed or an outdated version is present, install the latest Java Development Kit (JDK).

2. Install Eclipse or any other latest IDE:

Download the Eclipse installer and run it.
Select the option to install Eclipse for Java developers.
Open the Eclipse workbench.

3. Get the Codebase:

Clone the selenium-pom-factory-fluent-pattern repository from GitHub.

4. Set up the Project in Eclipse:

Open Eclipse and go to File -> Open Project from File System.
Browse the cloned project folder and open it.
Install the TestNG plugin for Eclipse by going to Help -> Eclipse Marketplace -> search for TestNG and Install it and Restart Eclipse after the installation.
The Maven plugin should be available by default. To verify, right-click on the project and check if there is an option called "Maven".

Possible issues:

If there is an error in the pom.xml file, try the following steps:
- Open the terminal and navigate to Users/<profile_name>/.m2.
- Run rm -r repository.
- Right-click on the project and select "Update project".
If there are errors in all import statements, try the following steps:
- Click on the src/main/java folder, go to Build Path, and remove it from the build path.
- Click on src -> main -> java, right-click, go to Build Path, and select "Use as Source Folder".
- Refresh the project.

5. Set up verification:

In the test package, right-click on any Java file and select "Run As TestNG Test".

Simple Example Test

Let's get started with a simple example of creating a test for flight booking using the page factory/fluent interface pattern.

Test Steps:

Go to www.cleartrip.com.
Select the trip type (One Way Trip).
Select the origin (Bangalore).
Select the destination (Delhi).
Select a date (Any random date).
Click on the "Search" button.

Refer to the image below to understand the page classes and test classes for different application pages:

Step 1: Create a Page Class

We need to create page objects for all the implemented pages in the ClearTrip site: HomePage.java, FlightResultsPage.java, HotelResultsPage.java, HotelsPage.java, and SignInPopup.java. These page classes will be used in the test classes to perform actions and assertions.

Let's consider the following scenario: On the home page, the user selects the flight booking option, enters valid details, and clicks the search button, which redirects them to the flight results page.

Example page class for the Home page (HomePage.java):



public class HomePage {
    private WebDriver driver;
    private WebDriverWait wait;

    @FindBy(xpath = "//nav[@class='row fieldRow tripType']")
    private WebElement tripType;

    @FindBy(id = "FromTag")
    private WebElement origin;

    // Other web elements and methods...

    public HomePage(WebDriver driver) {
        this.driver = driver;
        PageFactory.initElements(driver, this);
        wait = new WebDriverWait(driver, WaitTimes.TIME_5_SECONDS.getWaitTime());
    }


    // Methods for interacting with the elements and performing actions on the Home page...

    /**
     * This method launches the home page
     */
    public HomePage launch() {
        driver.get("https://www.cleartrip.com/");
        return this;
    }

    /**
     * This method selects the trip type
     *
     * @param tripTypeOption Value of Trip Type
     */
    public HomePage selectTripType(String tripTypeOption) {
        tripType.findElement(By.xpath("//input[@value='" + tripTypeOption + "']")).click();
        return this;
    }

    /**
     * This method selects the place of origin
     *
     * @param origin Value of Origin
     */
    public HomePage selectOrigin(String origin) {
        this.origin.clear();
        this.origin.sendKeys(origin);
        wait.until(ExpectedConditions.visibilityOfAllElements(originOptions));
        originOptions.get(Indices.INDEX_0.getIndexValue()).click();
        return this;
    }

    /**
     * This method clicks on the search button after
     */
    public FlightResultsPage clickSearchButton() {
        searchButton.click();
        return new FlightResultsPage(driver);
    }

    /**
     * This method clicks on the "Hotels" link in the home page
     */
    public HotelsPage clickHotelsLink() {
        hotelLink.click();
        return new HotelsPage(driver);
    }

    /**
     * This method clicks on the SignIn option which appears after clicking the "Your Trips" options
     */
    public SignInPopup clickSignInOption() {
        signIn.click();
        return new SignInPopup(driver);
    }

}

In the above example, we have created a page class for the ClearTrip Home page. It contains the necessary web elements and action methods associated with the Home page, initialized using the PageFactory model.

Please note that this is just an example, and you'll need to create similar page classes for other pages in the application.

Step 2: Create a TestBase Class

Create a Base Test Class which is extended in all the Test Classes. So The WebDriver is instantiated in such a way that it is static and only one instance will be created and shared across all the test classes. The Driver will only quit once all the tests are done.

TestBase.java



/**
 * Base Test Class which is extended Test Classes
 */
public abstract class TestBase {

    /**
     * Static WebDriver Instance
     */
    protected static WebDriver driver;


    public WebDriver getDriver() {
        return driver;
    }

    /**
     * Suite level setup method where the static instance of the WebDriver is instantiated
     */
    @BeforeSuite
    public static void initDriver() {

        WebDriverManager.chromedriver().setup();
    driver = new ChromeDriver();

    ChromeOptions options = new ChromeOptions();

        // The following statement sets the argument to disable notification in chrome when executing the tests
        options.addArguments("--disable-notifications");
        driver = new ChromeDriver(options);
        driver.manage().window().maximize();
    }


    /**
     * Suite level tear-down method to quit the WebDriver instance
     */
    @AfterSuite
    public void quitDriver() {
        driver.quit();
    }
}

Step 3: Create a Test Class

In test class, instances of the page classes will be created to access methods in the page classes. Thus, tests are decoupled from the pages/ elements under test. This makes the framework more maintainable.

So in this FlightBookingTest class we have to create the objects of HomePage, FlightResultsPage to access the methods available in those pages.

FlightBookingTest.java



/**
 * Class containing test for validating search-flight results
 */
public class FlightBookingTest extends TestBase {

    /**
     * Instance of HomePage class
     */
    private HomePage homePage;

    /**
     * Instance of FlightResultsPage class
     */
    private FlightResultsPage flightResultsPage;

    /**
     * Class level test-Setup method that intantiates {@link HomePage} and {@link FlightResultsPage}
     */
    @BeforeClass
    public void testClasSetup() {
        homePage = new HomePage(driver);
        flightResultsPage = new FlightResultsPage(driver);
    }

    /**
     * Test method for verifying whether search summary is rightly displayed when searching for flights
     */
    @Test
    public void testThatResultsAppearForAOneWayJourney(Method method) {

        ExtentTestManager.startTest(method.getName(), "Test method for verifying whether search summary is rightly displayed when searching for flights");

        ExtentTestManager.getTest().log(LogStatus.INFO, "Launching the browser");
        homePage.launch() 
            .selectTripType(TripTypes.ONE_WAY.getTripType())
            .selectOrigin(Cities.BANGALORE.getCity())
            .selectDestination(Cities.Delhi.getCity())
            .selectDate()
            .clickSearchButton();

        ExtentTestManager.getTest().log(LogStatus.INFO, "Asserting flight search summary details");
        Assert.assertTrue(
                flightResultsPage.isSearchSummaryAvailable(),
                "Search Summary is not available after clicking the Search Button when searching for flights");
    }


}

In the above example, we create a test class FlightBookingTest that uses the HomePage page class to perform actions on the ClearTrip website's Home page and then asserts or performs further actions on the Flight Results page.

Step 4: Run the Test

To run the test, right-click on the test class FlightBookingTest and select "Run As TestNG Test". This will execute the test steps and display the test results in the console.

Extent HTML report

Test Reports can be found in workingDir/ExtentReports/ExtentReportResults.html

Test Summary

Specific Test details

Conclusion

The framework provides an oranized structure for creating page classes, test classes, and generating interactive test reports using Extent Reports.

By following the installation steps and the provided example, you can set up the framework on your local system and start creating UI automation tests for web applications.

For a more detailed understanding of each component and the complete source code, please refer to the GitHub repository.

Integrating Newman API test collections with Jenkins and publish reports using Flock messenger

Abubakkar Sithik — Fri, 11 Jun 2021 15:01:20 +0000

In this article, we will be exploring about how to run Postman collections on Jenkins with Freestyle Project. And we will try to generate our test run reports using Allure Jenkins plugin and will publish the test run status notifications to Flock messenger.

Jenkins Setup
- Step1:Install NodeJS plugin
- Step2:Add Newman and Allure Packages
- Step3:Create a Newman job with freestyle.
How to Run

Jenkins Setup

Before we get started, you’ll need to install Jenkins on your machine. If you don’t have a setup Jenkins on your machine please follow this official documentation.

After installing Jenkins, we need to configure NodeJS on Jenkins.

Step1: Install NodeJS plugin :

Go to → Manage Jenkins → Select Manage Plugins → Select Available tab and Search Active Choice Plugin.

Install and restart the Jenkins for proper installation of the plugin.

Step2: Add Newman and Allure Packages in NodeJS

Open Jenkins: Manage Jenkins → Global Tool Configuration → NodeJS

Set your compatible node version. We can set multiple NodeJS versions for multiple applications.

Make sure you follow these steps:

Check the “Install automatically” option
Select compatible Node version (NodeJs-16.2.0)
Add required packages in the “Global Packages to Install” section In our case, add the following packages newman newman-reporter-allure
Save settings.

Step3: Create a Newman job with freestyle.

1. Create a Freestyle job to execute Newman collections scripts.
Open Jenkins → New Item → Enter any job name→ Choose Freestyle Project → Click on Save button

2. Parameterising the Jenkins build for different Newman environment
A build parameter allows us to pass data into our Jenkins jobs. Build parameters are a powerful way to make any Jenkins job more dynamic.So by using build parameters, we can pass any data we want. In our use case, we will be passing the Newman environment variables. Note: Follow this tutorial, If you want to get a Newman environment variables’ unique id.

Extended Choice Parameter Plug-In Installation:

Go to 'Manage Jenkins' from the Jenkins homepage.
Navigate to Manage Plugins
Change the tab to Available
Search for extended choice parameter
Click “Download now and install after restart”

General → Check “This project is parameterized” → Add parameter → Choice parameter → add respective choice values for parameterization [QA / Prod].

3.Source Code Management → Check “None”

4.Build environment → Check “Delete workspace before build starts” 5.Build environment → Check “Provide Node & npm bin/ folder to PATH” → select NodeJs name

6.Build → Add build step → Select Execute Shell → And Paste the below code. → Save Your config.

eg:- newman run [collections] -e [environment] -r allure --reporter-allure-export target/allure-results --suppress-exit-code 1

newman run "https://api.getpostman.com/collections/[collection_uid]?apikey=[apiKey]" \
    --environment "https://api.getpostman.com/environments/$server?apikey=[apiKey]" \ 
    -r allure --reporter-allure-export target/allure-results \ 
    -- suppress-exit-code 1

Note: If you want to run Newman via shell command using a unique collection id & API Key follow this tutorial.

Allure and Flock Plugin Installation

Go to 'Manage Jenkins' from the Jenkins homepage.
Navigate to Manage Plugins
Change the tab to Available
Search for Flock and Check the box
Search for Allure Report and Check the box
Click “Download now and install after restart”

7.Post Build Action → Allure Report → Add Path (“target/allure-results”)
Allure Reports will be generated after every successful builds and report files are placed under the given $PATH

8.Post Build Action → Send Flock Notification (add flock webhook URL)

Please follow this guide to generate a Webhook URL

Go to https://apps.flock.com/?source=launcherbar
Search for Jenkins
Install and Configure to generate webhook URL.

How to Run

Now we just have to build our job to execute our tests. And we can do so by clicking on the “Build with Parameters” button on the left side of the dashboard.

Go to Jenkins Job [jenkins_url/job/[#JobName]]
Click on Build with Parameter → Select the options that are required to be passed in the branch parameter [QA/ Prod] and click on → BUILD
Flock messenger would notify you about the Jenkins build status.
To view reports click on “View” in flock → Will be redirected to Jenkins [After successful authentication]→ Click on Allure Report

In this article we have seen how to use Jenkins to run Newman tests, generate & publish reports using Allure / Flock plugins. I hope you may have found this useful. Cheers!!

Automated Visual Regression Testing with BackstopJS

Abubakkar Sithik — Thu, 10 Jun 2021 11:24:06 +0000

What is Visual Regression Testing?

In a definition from more famous Wikipedia, Regression testing is re-running functional and non-functional tests to ensure that previously developed and tested software still performs after a change. In Visual Regression Testing, we're trying to run a test to verify the application has no visual difference from its former or less developed state. So in visual regression testing(VRT), most of the time we're not going to test some sort of functionality that we've built.

Typically, the main use-case of VRT comes in a few different ways. One of them is just doing updates for web app. When an application is getting an update(like upgrading internal libraries or security updates), we usually don't expect anything to be visually different. So, after an update application should look exactly the same and work exactly the same way before and after the update. So in VRT, we make sure that everything is working fine before the code goes live.

Another possible use case is doing production/live sanity testing. So, after the prod server code update, the application in the prod server should look exactly the same and work exactly the same way as dev or test environment.

While doing visual regression manually, we might be able to catch very blatant differences, but it's really difficult for us to catch subtle differences and also performing testing repeatedly. That's where a tool like BackstopJS helps us to automatically highlight the differences between the two screenshots that will make our testing a much easier.

BackstopJS
Backstop Features
BackstopJS workflow
BackstopJS Benefits
Installation
- BackstopJS Installation
BackstopJS project structure
Creating a visual regression test
- 1. Navigate to your local project
- 2. Initialize Backstop
- 3. Edit backstop.json
- Add viewports
- Add scenarios
- How to handle cookies / sessions in backstopJS
- 4. Create new reference screenshots
- 5. Run tests
- 6. Backstop approve

BackstopJS

BackstopJS is a framework that automates visual regression testing. This framework is written in JS and consists of the following tools:Puppeteer(headless chrome)+ ResembleJS(diff library). It uses a headless Chrome, in that way it's not actually opening up our Chrome browser and it's not taking screenshots that we can see. All we have to do is, write a script for simulating user scenarios and run backstopJS commands and it goes through and simulates user flows with headless chrome automatically. All we have to do is, run a simple command in the command-line tool so that it will take care of all the work for us.

Backstop Features

Renders with headless chrome
Simulates user interactions with simple JS / Puppeteer scripts
Browser reports with visual diffs, CLI reports, and JUnit reports
Easy to integrate with CI(Jenkins) and source control
Very easy to use, just 3 commands to do everything.

BackstopJS workflow

BackstopJS Benefits

"Reference (production env) vs Test (test/staging env)" Comparison
Multiple viewports support (desktop browsers, mobile browsers,..etc.)
Easy way to write UI tests (supports puppeteer scripts)
Inbuilt interactive and detailed reports
Easily way to scan our web application (backstop-crawl)

BackstopJS Installation

Check Node

The first thing you should do is check to see if you have a version of Node.js already installed. To do that in mac:
1. Open the Terminal & Enter `node - v` in the Terminal and press Enter
2. If you do have Node.js installed, it will output the version. Update to the latest version using `npm i -g npm`.
3. If you do not have Node.js installed, it will output something like `-bash: node: command not found` Continue with these instructions to install it. Go to https://nodejs.org/en/download/ You'll see download links for macOS. After Download, open the file & Go through the entire installation process. 
4. Enter node - v in the Terminal to verify that Node.js is installed correctly.

BackstopJS Installation

Global installation (recommended)

$ npm install -g backstopjs

BackstopJS project structure

Inorder to create a new visual automation testing project, First of all, we need to initialize Backstop. To do that run backtop init in the terminal.

mkdir ./sample_visualTestingProject
cd ./sample_visualTestingProject
backstop init

Now, the script will generate a new backstop config file backstop.json and a folder backstop_data. backstop.json has some pre-configured stuff in it. backstop_data folder has some file structure in place to save the screenshots it's gonna be generating.

The generated backstop.json file, should look something like this

{
  "id": "backstop_default",
  "viewports": [
    {
      "label": "phone",
      "width": 320,
      "height": 480
    },
    {
      "label": "tablet",
      "width": 1024,
      "height": 768
    }
  ],
  "onBeforeScript": "puppet/onBefore.js",
  "onReadyScript": "puppet/onReady.js",
  "scenarios": [
    {
      "label": "BackstopJS Homepage",
      "cookiePath": "backstop_data/engine_scripts/cookies.json",
      "url": "https://garris.github.io/BackstopJS/",
      "referenceUrl": "",
      "readyEvent": "",
      "readySelector": "",
      "delay": 0,
      "hideSelectors": [],
      "removeSelectors": [],
      "hoverSelector": "",
      "clickSelector": "",
      "postInteractionWait": 0,
      "selectors": [],
      "selectorExpansion": true,
      "expect": 0,
      "misMatchThreshold" : 0.1,
      "requireSameDimensions": true
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference",
    "bitmaps_test": "backstop_data/bitmaps_test",
    "engine_scripts": "backstop_data/engine_scripts",
    "html_report": "backstop_data/html_report",
    "ci_report": "backstop_data/ci_report"
  },
  "report": ["browser"],
  "engine": "puppeteer",
  "engineOptions": {
    "args": ["--no-sandbox"]
  },
  "asyncCaptureLimit": 5,
  "asyncCompareLimit": 50,
  "debug": false,
  "debugWindow": false
}

backstop.json is a configuration file, everything related to tests are defined in this file.

Lets start off with viewports.

"viewports": [
    {
      "label": "phone",
      "width": 320,
      "height": 480
    },
    {
      "label": "tablet",
      "width": 1024,
      "height": 768
    }
  ]

viewports are just a dimensions of the application site that we want to test /capture screenshots. The above example has one for phone and another one for tablet, maybe if we want to do desktop or whatever other dimensions are relevant for testing, we can simply add those dimenesion in viewports section.

In below section of backstop.json, we have a section for scenarios where we can define different pages on our application and all the scenarios/flows for that.

"scenarios": [
    {
      "label": "BackstopJS Homepage",
      "cookiePath": "backstop_data/engine_scripts/cookies.json",
      "url": "https://garris.github.io/BackstopJS/",
      "referenceUrl": "",
      "readyEvent": "",
      "readySelector": "",
      "delay": 0,
      "hideSelectors": [],
      "removeSelectors": [],
      "hoverSelector": "",
      "clickSelector": "",
      "postInteractionWait": 0,
      "selectors": [],
      "selectorExpansion": true,
      "expect": 0,
      "misMatchThreshold" : 0.1,
      "requireSameDimensions": true
    }
  ]

We have a label which is describing what this particular scenario is, and we have the URL which is the URL that we're testing, this is gonna be our production application URL and then the reference URL which is the baseline URL that we're testing it against. All these scenario properties are described here,

label                    // [required] Tag saved with your reference images
onBeforeScript           // Used to set up browser state e.g. cookies.
cookiePath               // import cookies in JSON format (available with default onBeforeScript see setting cookies below)
url                      // [required] The url of your app state
referenceUrl             // Specify a different state or environment when creating reference.
readyEvent               // Wait until this string has been logged to the console.
readySelector            // Wait until this selector exists before continuing.
delay                    // Wait for x milliseconds
hideSelectors            // Array of selectors set to visibility: hidden
removeSelectors          // Array of selectors set to display: none
onReadyScript            // After the above conditions are met -- use this script to modify UI state prior to screen shots e.g. hovers, clicks etc.
keyPressSelectors        // Takes array of selector and string values -- simulates multiple sequential keypress interactions.
hoverSelectors           // *Puppeteer only* takes array of selectors -- simulates multiple sequential hover interactions.
clickSelectors           // *Puppeteer only* takes array of selectors -- simulates multiple sequential click interactions.
postInteractionWait      // Wait for a selector after interacting with hoverSelector or clickSelector (optionally accepts wait time in ms. Idea for use with a click or hover element transition. available with default onReadyScript)
scrollToSelector         // Scrolls the specified DOM element into view prior to screen shot (available with default onReadyScript)
selectors                // Array of selectors to capture. Defaults to document if omitted. Use "viewport" to capture the viewport size. See Targeting elements in the next section for more info...
viewports                // An array of screen size objects your DOM will be tested against. This configuration will override the viewports property assigned at the config root.

for more details refer this doc.

Creating a visual regression test

Now, let's get started with the simple user flow:

Land on page 1 -> do some action (click a link/button) -> go to page 2

Scenarios:

Scenario 1: land on page 1, BackstopJS takes screenshot for page 1
Scenario 2: from page 1, do some action and go to page 2, BackstopJS takes screenshot for page 2

BackstopJS Flow

Navigate to your local project
Initialize Backstop: backstop init
Edit your backstop.json file
- Add viewports
- Add scenarios
Create new reference screenshots: backstop reference
Run tests: backstop test
Approve tests: backstop approve

1. Navigate to your local project

After installing BackstopJS, create (or) navigate to project folder,

mkdir ./sample_visualTestingProject
cd ./sample_visualTestingProject

2. Initialize Backstop

Intialize backstop setup by running the following command. This command will generate a new backstop config file backstop.json and a folder backstop_data. This is just an one time command, if backstop files are already there, we need not to run this command again.

backstop init

3. Edit backstop.json

3.1 Add viewports

open backstop.json in any editor, and set viewports for desktop and mobile chrome,

"viewports": [
    {
      "label": "Desktop Chrome",
      "width": 1280,
      "height": 800
    },
    {
      "label": "Mobile Chrome",
      "width": 375,
      "height": 812
    }
  ]

3.2 Add scenarios

Scenario 1: land on page 1, BackstopJS takes screenshot for page 1

Let's use this walmart product page as base page. So after user landing on this page, we have to take screenshot of this page using backstopJS.

    {
      "label": "walmartProductPage",
      "url": "https://www.walmart.com/ip/Super-Mario-Odyssey-Nintendo-Switch/56011600",
      "referenceUrl": "https://www.walmart.com/ip/Super-Mario-Odyssey-Nintendo-Switch/56011600",
      "delay": 5000
    }

label: name of your scenario
url: Test URL
referenceUrl: baseline URL (since we don't have any development environment url, using same url here for demo purpose.)
delay: It allows you to set a time to wait for the page to load before assuming it'll be ready to test.

Scenario 2: from page 1, do some action and go to page 2, BackstopJS takes screenshot for page 2

In this scenario, we are moving from product page -> cart page by clicking on add to cart button button.prod-ProductCTA--primary. After landing on this cart page, we have to take screenshot of this page using backstopJS.

  {
    "label": "walmartCartPage",
    "url": "https://www.walmart.com/ip/Super-Mario-Odyssey-Nintendo-Switch/56011600",
    "referenceUrl": "https://www.walmart.com/ip/Super-Mario-Odyssey-Nintendo-Switch/56011600",
    "readySelector": "button.prod-ProductCTA--primary",
    "clickSelector": "button.prod-ProductCTA--primary",
    "delay": 5000
  }

readySelector - Making Backstop to wait until a particular element (defined by CSS selector) is appearing before starting.
clickSelector - Making Backstop to click a particular element (defined by CSS selector)

Final backstop.json file should look like this for the given scenario,

{
  "id": "sample_project",
  "viewports": [
    {
      "label": "Desktop Chrome",
      "width": 1280,
      "height": 800
    },
    {
      "label": "Mobile Chrome",
      "width": 375,
      "height": 812
    }
  ],
  "onBeforeScript": "puppet/onBefore.js",
  "onReadyScript": "puppet/onReady.js",
  "scenarios": [
    {
      "label": "walmartProductPage",
      "url": "https://www.walmart.com/ip/Super-Mario-Odyssey-Nintendo-Switch/56011600",
      "referenceUrl": "https://www.walmart.com/ip/Super-Mario-Odyssey-Nintendo-Switch/56011600",
      "delay": 5000
    },
    {
      "label": "walmartCartPage",
      "url": "https://www.walmart.com/ip/Super-Mario-Odyssey-Nintendo-Switch/56011600",
      "referenceUrl": "https://www.walmart.com/ip/Super-Mario-Odyssey-Nintendo-Switch/56011600",
      "readySelector": "button.prod-ProductCTA--primary",
      "clickSelector": "button.prod-ProductCTA--primary",
      "delay": 5000
    }
  ],
  "paths": {
    "bitmaps_reference": "backstop_data/bitmaps_reference",
    "bitmaps_test": "backstop_data/bitmaps_test",
    "engine_scripts": "backstop_data/engine_scripts",
    "html_report": "backstop_data/html_report",
    "json_report": "backstop_data/json_report",
    "ci_report": "backstop_data/ci_report"
  },
  "report": ["browser"],
  "engine": "puppeteer",
  "engineOptions": {
    "args": ["--no-sandbox"]
  },
  "asyncCaptureLimit": 5,
  "asyncCompareLimit": 50,
  "debug": false,
  "debugWindow": false
}

In this demo project other than viewports & scenarios, we don't really have to change anything else. Based on the requirement, We can change the all the other variables if needed. For more details about backstop.json properties, refer this doc.

How to handle cookies / sessions in backstopJS

In some scenarios, we might are try to access urls without repeating any login actions. When an application url has a cookie dependencies, backstopJS provides us a way to import cookies through json files. In such scenarios, we have to add the following backstop property in backstop.json file

"cookiePath": "backstop_data/engine_scripts/cookies.json"

In backstopJS, it is possible to first run a Puppeteer script that logins into application portal, then save the session cookies in a file. Now a subsequent scenarios can read those json file to load cookies and proceeds to do some action - all without having to log in again.

To run a custom script, add this step "onReadyScript": "puppet/getCookies.js" in backstop.json file to execute custom puppeteer script to handle cookies/session dynamically. You can find all custom puppeteer scripts in this location workingdir > backstop_data > engine_scripts > puppet.

The following getCookies.js script, grabs cookies by logging into the app and stores the cookies in mentioned cookiePath location.

const fs = require('fs');
const  cookiePath = "backstop_data/engine_scripts/cookies.json";

module.exports = async (page, scenario, vp) => {

    console.log('SCENARIO > ' + scenario.label);

    console.log("Closing cookie consent");
    await page.waitForSelector('button.form__button.form__button--green.cookie-button');
    await page.click('button.form__button.form__button--green.cookie-button');

   // Waits until the `email & password` meta element is rendered
    await page.waitForSelector('input[name="email"]');
    await page.waitForSelector('input[name="password"]');

    await page.type('input[name="email"]', 'userEmail@email.com',{delay: 5});
    await page.type('input[name="password"]', 'Test1234!',{delay: 5});

    console.log("Clicking Submit");
    await page.waitForSelector('button[type='login']');
    await page.click('button[type='login']');

    await page.waitForNavigation();

    const cookies = await page.cookies();

    console.log("The cookie is:", cookies);

    fs.writeFile(cookiePath, JSON.stringify(cookies, null, 2), function(err) {
        if (err) throw err;
        console.log('completed write of cookies');
    });
};

4. Create reference screenshots: `backstop reference`

After saving the backstop.json file, switch back to command line, and start with a executing Backstop reference command backstop reference.

This command is gonna generate the initial reference screenshots. It will generate screenshots for the different view ports for the given scenarios, and then once that's done, we can see the generated screenshot images in workingDir > backstop_data > bitmaps_reference.

If there's any failure, we could see the errors in command-line window. Modify the backstop.json & rerun the reference command until getting a successful run.

5. Run tests: `backstop test`

Once the reference images are generated successfully, we have to run this command backstop test in command-line. Now backstopJS will generate screenshots of test application first and after that it will generate a report, by comparing them(reference vs test).

Lets just go through the report,

We can take a look at the reference and see the screenshot and kind of see what it looks like.
We can jump over to the test, see what that looks like.
We can also look at the DIFF to see everything highlighted that's different between the two, and then we can actually go to the scrubber and we can see a comparison of the two versions and scrub them back and forth.

6. Backstop approve: `backstop approve`

If the test we ran looks good, then go ahead and approve it. Approving changes will update your reference files with the results from your last test. Future tests are compared against your most recent approved test screenshots.

What we are trying to kind of learn here is that we can run BackstopJS on our computer in a pretty easy & quick way to automate our most of manual visual regression tests. In this demo project, we have explored the very simple test scenario to understand the BackstopJS. In real time scenarios, we might have to use puppeteer JS scripts to handle a complicated scenarios.

DEV Community: Abubakkar Sithik

Running Postman Collections in Jenkins

The Problem

The Solution

Prerequisites

Implementation Details

1. Pipeline Configuration

2. Newman Integration

3. Collection Execution

4. HTML Report Generation

5. Email Notifications

Setting Up the Pipeline

1. Required Plugins

2. NodeJS Configuration

3. Credentials Setup

4. Email Configuration

Pipeline Configuration

1. Create Pipeline Job

2. Configure Pipeline

3. Environment Configuration

Postman Setup

1. Collection Preparation

2. Environment Setup

First Run

Conclusion

Cleaning Up Stale Branches to Speed Up CI/CD and Reduce GitHub API Load

The Problem: Too Many Stale Branches

The Solution: Automating Stale Branch Cleanup

The Script:

How It Works:

Lessons Learned & Best Practices

1️⃣ Automate but Review First

2️⃣ Protect Important Branches

3️⃣ Schedule Regular Cleanup

4️⃣ Use GitHub API Efficiently

Conclusion

Automating Mirroring Between Bitbucket and GitHub

Overview

Steps to Automate Mirroring

1. GitHub Repository Setup

2. Bitbucket Pipelines Configuration

3. SSH Key Generation (Bitbucket)

4. Known Hosts Configuration (Bitbucket)

5. Deploy Key Configuration (GitHub)

6. Access Key Configuration (Bitbucket)

7. Pipeline Configuration (Bitbucket)

Conclusion

How to Build a Selenium UI Testing Framework

Selenium UI Testing Automation Framework

Framework Details

Test Application

Framework - What and Why?

Project Structure

Key features of the framework:

Packages

Main Package

Test Package

Reports

Installation

Steps to follow to set up Selenium in the local system:

Simple Example Test

Step 1: Create a Page Class

Step 2: Create a TestBase Class

Step 3: Create a Test Class

Step 4: Run the Test

Extent HTML report

Conclusion

Integrating Newman API test collections with Jenkins and publish reports using Flock messenger

Contents

Jenkins Setup

Step1: Install NodeJS plugin :

Step2: Add Newman and Allure Packages in NodeJS

Step3: Create a Newman job with freestyle.

How to Run

Automated Visual Regression Testing with BackstopJS

What is Visual Regression Testing?

Contents

BackstopJS

Backstop Features

BackstopJS workflow

4. Create reference screenshots: `backstop reference`

5. Run tests: `backstop test`

6. Backstop approve: `backstop approve`