DEV Community: Akshay C A

Complete CI/CD pipeline

Akshay C A — Thu, 24 Feb 2022 18:50:16 +0000

A CI/CD pipeline is a series of steps that must be performed in order to deliver a new version of the software. Continuous integration/continuous delivery (CI/CD) pipelines are a practice focused on improving software delivery using either a DevOps or site reliability engineering (SRE) approach.

I've created a simple CI/CD pipeline with -

GitHub for SCM.
Jenkins.
Docker.
Docker Hub.
AWS EC2 Instances.
AWS EKS service.

Let's begin!

Step 1

Create an EC2 instance to run Jenkins.

I'm going to create an AWS EC2 instance and install docker in it to run Jenkins.

It is best practice for a user to access Jenkins with limited permission, so let's create a Jenkins user for this purpose.

Once this Jenkins container is up, edit the inbound rules to expose port 8080.

Let's finish up the initial settings

You need to make the data persist even after you restart the server.
So, stop the container, re-run docker the volume defined.

Next, provide the required access to docker.

So, we now have Jenkins up and running.

Step 2

Push your code to GitHub.

For this project, we will use the Java Maven App with a Postgres database.

You can clone my project for your reference - Repo Link

Here is the list of tasks that needs to be performed by Jenkins -

Build the solution and increment the version number.
Build the Docker Image using the Dockerfile and push the Image to the DockerHub.
Trigger the automation tests.
Deploy the App to a K8s cluster.
Commit version update to SCM.

Step 3

Create a simple pipeline project in Jenkins.

Set GitHub hook trigger for GITScm polling
Provide the Git Repository URL and credentials
Script Path as Jenkinsfile

Step 4

Create Jenkins shared library.

Jenkins Shared Libraries are written in Groovy and allow you to create common sets of logic, and share that among teams/projects/organizations.

I've created one for this project - Repo Link

Update your Jenkins file to read the shared library.

Increment version-

Build the Jar-

Build the Docker Image and Push it to the DockerHub -

Now if everything is configured properly, whenever you push your code to the master. Jenkins should trigger the pipeline and perform the below actions.

Clone the git repository
Increment the version number
Build the Jar file
Build Docker Image
Push the Image to Docker Hub

Step 5

Deploy to K8s cluster

Now we are supposed to trigger the automation tests. Once all the tests are passed and all the checklists are cleared, we can deploy our App to the production.

Install Kubectl command-line tool inside Jenkins container.
Install aws-iam-authenticator tool inside Jenkins container.

I've create an EKS cluster called demo-cluster.

Create Kubeconfig file to connect to EKS cluster.

contains all the necessary for authentication

Add AWS credentials on Jenkins for AWS account authentication

Adjust Jenkinsfile to configure EKS cluster deployment.

The last thing which you need to do once the deployment is successful is to commit the version update to the code repository.

The pipeline is now complete.

Now let's run it !!!

Push some changes to the Master branch our pipeline will be triggered.

Check the logs

Now let's connect to the EKS cluster from our local machine to verify.

Thank you!
Please comment for feedback.
Happy learning :)

Overview of our CI/CD pipeline -

Build Test Report Dashboard using MERN stack

Akshay C A — Tue, 04 Jan 2022 16:35:34 +0000

A test report dashboard is an organized summary of results. It is created and used to help stakeholders (product managers, analysts, testing teams, and developers) understand product quality and decide whether a product, feature or defect resolution is on track for release.

The idea is to build a Dashboard that quickly encapsulates test results from browser UI tests, windows UI tests, API tests, performance tests, etc., performed by a particular build.

I used MongoDB because its flexible schema makes it easy to evolve and store data. React and Express.js for building the web application and API.

So, the different testing frameworks would make the REST API call with the test results in JSON format to our application as soon as the test execution is completed.

Our App which will be running on a server would store this data and display it to all the stakeholders in real-time.

Let's get started !!

You can clone my code repository for GitHub for your reference : Link

Step 1

Prerequisite:
You need Docker installed on your machine.

You need mongo and mongo-express. So, create a docker-compose.yml file, add the below content.

version: '3'
services:
  mongodb:
    image: mongo
    ports:
     - 27017:27017
    environment:
     - MONGO_INITDB_ROOT_USERNAME=admin
     - MONGO_INITDB_ROOT_PASSWORD=password
    volumes:
     - mongo-data:/data/db
  mongo-express:
    image: mongo-express
    ports:
     - 8081:8081
    environment:
     - ME_CONFIG_MONGODB_ADMINUSERNAME=admin
     - ME_CONFIG_MONGODB_ADMINPASSWORD=password
     - ME_CONFIG_MONGODB_SERVER=mongodb
volumes:
  mongo-data:
    driver: local

Run the Docker compose command -

$ docker-compose -f docker-compose.yml up

You should be able to access it on localhost port- 8081.
Go ahead and create the database and name it DashboardApp

Step 2

Prerequisite:
You need Node installed on your machine.

Create the React application by running this command
npx create-react-app <app-name>
Now navigate into the App and create the backend folder.
Inside this folder, we will create the backend
npm init -y.
So, that it connects to the MongoDB then we will come back and write the React later.

We will install these dependencies for our backend.
npm install express cors mongoose dotenv

Create the server.js file to connect to the database and the .env file to store the environment variables.

Now you can start the server and the console should look something like this -

Step 3

Now let's create the database schema. Create a new directory called models.
Add all the different schemas you want to create in the model.js files.
Once this is done, we need to add the API endpoints routes to perform the CRUD operations.
Inside the backend folder, create another folder called routes and the CRUD operations code in it.

You can test the server by making an API call.

Step 4

Now it's time to build the front end using React.
You also npm install axios, bootstrap, react-bootstrap, react-icons, react-router-dom:5.0.0 and react-scripts:4.0.3.

You need to edit the default template provided by React in index.html, index.js, and in App.js files

You use components to tell what we want to see on the screen.
So, create a folder called components inside the src folder.
Create the components files or projects as per your project needs.

Once, this is done you can start the Dashboard by running the npm start command.
The App should start running at localhost port- 3000.

Added some more data into the database and the Dashboard UI should look something like this.

Step 5

Now, let's Dockerize our Dashboard application. So, it will be easy to start our app or run on only server easily.

We need to create a Dockerfile for the server and the client. The Dockerfile essentially contains the build instructions to build the image.

And it using the docker build command

$ docker build -t <image-name> .

To run our entire application together, i.e run all containers parallelly, we need to configure the docker-compose file.
So, I will be editing the existing docker-compose file.

To start the entire application services we need to run the following command:

$ docker-compose up

Note: You need to update the .env file in the backend since we are running the services within the docker container.
i.e MONGO_URI=mongodb://mongodb:27017/DashboardApp

You can access the application at localhost port- 3000.

Well, that's it!

You can run this Dashboard App on your machine by following these steps

Clone the repo

$ git clone https://github.com/akshayca/Dashboard.git
$ cd Dashboard

Start the app

Go to the project directory and run

$ docker-compose up --build

The app will start running at localhost:3000

Let me know if you need any help :)

Here are the links which you might find useful:

Learn the MERN Stack
Dockerizing a MERN Stack Web Application

Thank you!

SoapUi Load Test reporting Dashboard

Akshay C A — Sun, 10 Oct 2021 09:59:30 +0000

In this post I'd like to show you how to create the Load Test results dashboard using plotly and Dash.
Assuming that you use the SoapUi opensource version to run your load tests, you know there are very limited features to -

Archive the test results.
Analyze how the system is performing with respect to the previous builds.
Dashboard with graphical representation to show the performance of the system over time and load.

Here is the solution which I created to overcome these limitations.

Since we know a .csv file is generated with the statistics when we run the load test from the command prompt.

I used Python with Pandas to filter out the unwanted data and archive the results in the database using pymssql.

Now, I've my results archived in the database, I used the Dash and plotly to create a dashboard and hosted it on a server which can be accessible to the stakeholders.

So, Let's get Started !

Code Repository - https://github.com/akshayca/SoapUiLoadTestDashboard

Step 1 - Execution

Create a runLoadTest.bat file to run all your Load Tests from command prompt
I've used the calculator example project.

Below is the project structure, I've placed my example-soapui-project.xml in the TestCase folder -

the execution command would look something like this -

>set projectRootFolder=C:\SoapUiLoadTest\TestCase\

>set ReportFoler=C:\SoapUiLoadTest\TestResults\

>set limit=15
>set thread=10

>loadtestrunner.bat -s"CalculatorSoap TestSuite" -c"Add TestCase"  -l"LoadTestAdd" -m%limit% -n%thread% -r -f%ReportFoler% "%projectRootFolder%example-soapui-project.xml"

Run the runLoadTest.bat, execution will start and the reports will be generated with the current timestamp.

Step 2 - Data Archiving and Reporting

Create a database called 'SoapUILoadTestResults' to archive the results -

Run the createDashboard.py

Note: You need to pip install all the imports used for this project and have bWLwgP.css file in the assets folder.

Once the execution is completed, you can the check the data archived in the database.

Dash will be running on http://127.0.0.1:8050/

Once you have enough data on your dashboard would look something like this -

Code repository - https://github.com/akshayca/SoapUiLoadTestDashboard

Let me know if you need any help with this :)