Scaling Build Time of Open Source projects using Harness CI

Dhrubajyoti Chakraborty — Mon, 13 Mar 2023 19:41:18 +0000

In the world of software development, open-source projects play a critical role in shaping the industry. However, as these projects grow in complexity and size, so does their build time, which can make it challenging to test and release software on schedule.

While there are various Continuous Integration (CI) tools available to tackle this problem, I decided to try building some of the most well-known open-source projects on the Harness CI module. In this article, I will share my experience and provide a comparison of the build time with other CI tools available in the market.

Harness & Continuous Integration

Harness is a prominent player in the world of Continuous Integration and Continuous Deployment (CI/CD) platforms, offering a comprehensive suite of tools designed to help organizations streamline their software deployment and integration processes.

One of their most recent additions is the recent release of its Hosted Builds Solution for Harness CI which provides a new way for developers to scale their build times by leveraging remote infrastructure managed by Harness. This offering provides a dependable and efficient approach for projects to optimize their testing and software release processes by reducing build times.

Harness CI & Hosted Builds Architecture

Harness Cloud offers a robust and efficient solution for running builds on infrastructure hosted by Harnesss on cloud. By leveraging the power of Harness-Hosted Builds, developers can run builds in isolation and scale them as per requirement.

This solution can potentially become the game-changer for developers as it takes care of maintaining and upgrading build infrastructure, allowing teams to focus on building software without worrying about the underlying infrastructure.

One of the biggest advantages of Harness Cloud is the availability of free monthly credits for up to 2,000 build minutes, making it a cost-effective option for small and large-scale projects. Harness Cloud also offers starter pipelines for different programming languages, which can help accelerate the development process.

Underlying Architecture of Pipeline executions on Harness Cloud

When a pipeline is built using Harness Cloud, each stage of the pipeline is executed on a dedicated, ephemeral virtual machine (VM).

Within each stage, the steps are executed on the VM and can easily share information through the underlying filesystem. You can choose to run your CI steps directly on the VM or in a Docker container, depending on your project's needs.

One significant advantage of using Harness Cloud is that once a stage is complete, the VM automatically shuts down. This helps to optimize resource usage and avoid any unnecessary costs associated with running idle VMs.

To learn more about the architecture of the Harness Platform refer to their documentation here.

Replicating workflows of OSS projects in Harness CI

The task of replicating or migrating workflows from one tool to another can be quite daunting, especially for those who are new to the process. Thankfully, the Harness CI community has released various migration guides in the Harness Developer Hub and on Github to simplify this task.

If you're looking to migrate your CI workflow to Harness CI, these guides can be incredibly helpful.

Check out the migration guides here: Click Here

In this article, we will be exploring how to build two well-known open-source projects hosted on Github - GraphQL.js by GraphQL and Signal-Android by Signal App - using Harness CI.

Check out the codebase of GraphQL-JS here

Check out the codebase of Signal-Android here

GraphQL-JS is the JavaScript reference implementation for GraphQL, a query language for APIs created by Facebook. See more complete documentation at https://graphql.org/ and https://graphql.org/graphql-js/.

Signal is a simple, powerful, and secure messenger. Signal uses your phone's data connection (WiFi/3G/4G/5G) to communicate securely. Millions of people use Signal every day for free and instantaneous communication anywhere in the world. The project codebase is based on Java & Kotlin. Check out the project documentation wiki here

Migrating the existing workflow to Harness CI

As software development continues to evolve, so do the tools and services that developers use to build and deploy applications. In this case, both Graphql-JS and Sihgnal-Android use GitHub Actions as the CI/CD automation tool. To ensure a seamless transition when migrating workflows from GHA or any other CI tool to Harness CI, there are several important steps to consider. These include:

Assess the current workflow's complexity and understand how it operates.
Determine the number of jobs or steps required to complete the workflow.
Identify any dependencies that might exist between the various components of the workflow.
Determine the deployment and infrastructure targets that are required to support the workflow.
Configure any integrations necessary and set up the required environment for Harness CI.
Thoroughly test and validate the new pipelines to ensure they are performing as expected and producing the desired outcomes.

To begin the migration process, we need to follow the standard guidelines and activate the Github Actions workflow by enabling Actions in our forked repository. The first step is to validate the Github Actions workflow which involves verifying the syntax and structure of the workflow YAML file located under the .github/ directory.

Next, we convert the YAML file into a Harness CI workflow, which is stored in YAML format under the .harness/ directory. This involves mapping the corresponding jobs and actions in the GitHub Actions workflow to equivalent stages & steps in Harness CI. This is where we ensure the environment and variables along with secrets are setup & configured correctly in Harness CI.

Once the migration is complete and the workflows are successfully migrated, we can trigger pipeline execution manually or through various Git events such as pull request (PR) open, PR merge, etc. Harness CI provides several other options as well for triggering pipelines, including webhooks, scheduled triggers, and integrations with various CI/CD tools.

For more detailed information on the different triggering options available in Harness CI, refer to the documentation here.

The migrated workflows and their corresponding pipeline files for GraphQL-JS and Signal-Android are available in the .harness/ directory of the forked GitHub repository.

Testing & Validating the Pipeline & Build Time

The pipeline for testing and validating the GraphQL-JS and Signal-Android project can be manually executed or triggered by git events or customized use cases. The infrastructure used for the pipeline is Hosted Builds, with the respective stages and steps configured in the execution section. Working directories and default directories are specified in the Overview section, while the Advanced section allows for the implementation of failure and looping strategies if necessary.

After executing the workflows of GraphQL-JS & Signal-Android we receive the following observations.

Project: GraphQL-JS

Build Time in GitHub Actions: 4 hours 30 minutes (Average)

Build Time in Harness CI: 56 minutes (Average)

Project: Signal-Android

Build Time in GitHub Actions: 26 minutes (Average)

Build Time in Harness CI: 12 minutes (Average)

Project	Build Time in Harness	Build Time in Github Actions
GraphQL-JS	~56 minutes	~4 hours 30 minutes
Signal-Android	~12 minutes	~26 minutes

We can observe that the build time in Harness CI is significantly faster than that in GitHub Actions. Specifically, the average build time in Harness comparatively faster than in GitHub Actions.

This highlights Harness CI's capability of executing the same build at the same time with improved efficiency, potentially revolutionizing Continuous Integration in the software industry.

Object Specifier (Yolo-Coco)

Dhrubajyoti Chakraborty — Tue, 08 Sep 2020 08:44:52 +0000

codewdhruv / obj-specifier

AI model to detect objects from a given input picture

Object Specifier

Dataset

The dataset used can be downloaded here - Click to Download

This is ready to use data with weights and configuration along with coco names to detect objects with YOLO algorithm.

Prerequisites

All the dependencies and required libraries are included in the file requirements.txt See here

Installation

Clone the repo

$ git clone https://github.com/dhrubajyoti89/obj-specifier.git

Change your directory to the cloned repo and create a Python virtual environment named 'test'

$ mkvirtualenv test

Now, run the following command in your Terminal/Command Prompt to install the libraries required

$ pip3 install -r requirements.txt

Working

Open terminal. Go into the cloned project directory folder and type the following command:

$ python3 yolo.py --dataset dataset

To detect objects in an image type the following command:

$ python3 yolo.py -i "images/1.jpg" -y "yolo-coco"

Results

Our model gave 81% accuracy for Object Detection after training

Project Demo

Dev Link

Feel free to mail…

View on GitHub

Built it using yolo based object detector model. The dataset was made custom using object recognition API and Kaggle datasets of real large datasets of object images(yolo-coco). Used Python script with numpy, argparse, os libraries as the backend.

DEV Community: Dhrubajyoti Chakraborty