DEV Community: Adnan Latif

From 41 Minutes to 8 Minutes: How I Made Our CI/CD Pipeline 5x Faster

Adnan Latif — Mon, 16 Dec 2024 18:23:11 +0000

From 41 Minutes to 8 Minutes: How I Made Our CI/CD Pipeline 5x Faster

Introduction

In the world of software development, time is everything. Continuous Integration/Continuous Deployment pipelines speed the process up, but sometimes it’s the pipeline that makes the process slow down, ironically. This was what brought me to my latest problem when our Jenkins pipeline grew to an unmanageably long 41 minutes per build.

Determined to eliminate this inefficiency, I analyzed, optimized, and transformed our pipeline from a whopping 41 minutes down to 8 minutes — a 5x improvement! In this article, I’ll walk you through the issues I encountered, the solutions I implemented, and the strategies you can use to supercharge your own pipeline.

The Problem

Our CI/CD pipeline handled the following tasks for both backend and frontend:

Code Checkout
Static Code Analysis: ESLint, SonarQube
Unit Testing
Docker Image Build and Push
Staging Deployment
Manual Approval and Production Deployment

At first glance, the pipeline appeared robust, but some issues came into view:

Bloated Docker Build Context
The build context — all the files sent into Docker during an image build — had grown to 1.5GB and was taking a really long time to build.
Installing Dependencies Redundantly
Every stage in the pipeline had to reinstall the npm dependencies from scratch, thus adding unnecessary delays.
Poor Docker Image Management
Docker images were rebuilt and pushed to the registry, even when no changes had occurred.
No Parallel Execution
Similarly, all tasks, such as static code analysis or testing, were run sequentially.
Manual Deployment Steps
Since this involved updating AWS ECS task definitions manually, deployment of the backend was time-consuming and prone to human error.

The Solutions

Here’s How I Transformed the Pipeline for a 5x Optimization.

Reduce the size of the Docker Build Context

The Docker build context was unnecessarily large due to unfiltered project directories. We can use .dockerignore file to exclude certain files such as node_modules, logs etc.

Key File: .dockerignore

node_modules  
*.log  
dist  
coverage  
test-results

Impact:
Reduced the build context size from 1.5GB to ~10MB, reducing the transfer time from 30 minutes to <1 minute.

Dependency Caching

Every stage was using npm install. I replaced it with npm ci for reproducibility and activated caching in Jenkins.

Command Update:

npm ci --cache ~/.npm

Impact:
Reduced dependency installation time from 3–4 minutes per stage down to <20 seconds.

Improve Docker Image Handling

Previously, the pipeline would rebuild and push Docker images irrespective of changes. I added the logic to compare the hash of local and remote images, doing a push only if the image changed.

Updated Logic:

def remoteImageHash = sh(returnStdout: true, script: "docker inspect --format='{{.Id}}' $DOCKER_IMAGE:$DOCKER_TAG || echo ''").trim()
def localImageHash = sh(returnStdout: true, script: "docker images --no-trunc -q $DOCKER_IMAGE:$DOCKER_TAG").trim()

if (localImageHash != remoteImageHash) {
    sh 'docker push $DOCKER_IMAGE:$DOCKER_TAG'
} else {
    echo "Image has not changed; skipping push."
}

Impact:
Avoided unnecessary pushes, saving 3–5 minutes per build.

Run Static Analysis and Testing in Parallel

I extended the Jenkins pipeline to make use of the parallel directive so that tasks like ESLint, SonarQube analysis, and unit tests could proceed simultaneously.

Updated Pipeline:

stage('Static Code Analysis') {
    parallel {
        stage('Frontend ESLint') {
            steps {
                sh 'npm run lint'
            }
        }
        stage('Backend SonarQube') {
            steps {
                withSonarQubeEnv() {
                    sh 'sonar-scanner'
                }
            }
        }
    }
}

Impact:
Reduced static analysis and testing time by 50%.

Automatic Backend Deployment

Manual updates to AWS ECS task definitions were time-consuming and error-prone. I automated this step using the AWS CLI.

Automated Script:

def taskDefinitionJson = """
{
    "family": "$ECS_TASK_DEFINITION_NAME",
    "containerDefinitions": [
        {
            "name": "backend",
            "image": "$DOCKER_IMAGE:$DOCKER_TAG",
            "memory": 512,
            "cpu": 256,
            "essential": true
        }
    ]
}
"""
sh "echo '${taskDefinitionJson}' > task-definition.json"
sh "aws ecs register-task-definition --cli-input-json file://task-definition.json --region $AWS_REGION"
sh "aws ecs update-service --cluster $ECS_CLUSTER_NAME --service $ECS_SERVICE_NAME --task-definition $ECS_TASK_DEFINITION_NAME --region $AWS_REGION"

Impact:
Streamlined deployments, shaving off 5 minutes.

Results

After these optimizations, the pipeline time came down from 41 minutes to just 8 minutes — a 5x improvement. Here’s a detailed comparison:

Lessons Learned

Logs Are Your Best Friend: Analyze logs to pinpoint bottlenecks.
Caching Saves the Day: Effective use of caching can drastically cut build times.
Run Tasks in Parallel: Use parallel execution for immediate time savings.
Exclude Irrelevant Files: A .dockerignore file can significantly boost performance.
Automate Repetitive Tasks: Automation eliminates errors and speeds up workflows.

Conclusion

Optimizing a CI/CD pipeline was an eye-opening experience. Targeting key bottlenecks and implementing strategic changes transformed a 41-minute chore into an 8-minute powerhouse. The result? Faster deployments, happier developers, and more time to focus on features.

If you’re struggling with a slow pipeline, start by identifying bottlenecks, leverage caching, parallelize tasks, and automate repetitive steps. Even small tweaks can lead to massive gains.

How much time have you saved by optimizing your CI/CD pipeline? Share your experiences and tips in the comments below!

From Confusion to Clarity: Understanding ‘this’ in JavaScript

Adnan Latif — Tue, 30 Jan 2024 08:27:28 +0000

From Confusion to Clarity: Understanding ‘this’ in JavaScript

In the world of JavaScript, the this keyword often stands as a mystery, perplexing many new developers. Understanding this is crucial for writing efficient and bug-free code. In this article, we'll embark on a journey to demystify this and unravel its secrets in simple terms that even a novice developer can grasp.

What is “this”? In JavaScript, this is a special keyword that refers to the context in which a function is called. This context can vary based on how and where a function is invoked. Think of this as a placeholder that dynamically represents different values depending on the situation.

Understanding this in Different Contexts:
Inside a function body in JavaScript, the behavior of the this keyword depends on how the function is called.
Regular Functions:
In non-arrow functions, this is determined by the caller of the function.
If the function is called as a method of an object, this refers to that object.
If the function is called without a context, this defaults to the global object (window in browsers, global in Node.js), or undefined in strict mode.
Arrow Functions:
Arrow functions do not have their own this binding. Instead, they inherit this from the enclosing lexical scope.
In other words, this inside an arrow function refers to the this value of the enclosing scope where the arrow function was defined.

Global Context: When a function is called without any explicit context, this refers to the global object. In web browsers, this is usually the window object.

Object Method: When a function is called as a method of an object, this points to the object that owns the function.

Arrow Functions: Arrow functions do not have their own this binding and inherit this from the enclosing lexical scope.

Constructor Function: Inside a constructor function, this refers to the specific instance of the object being created.

Event Handlers: In event handler functions, this typically points to the DOM element that triggered the event.

Node.js Environment: In Node.js, this behaves differently compared to the browser environment. In most cases, this in Node.js modules refers to the module.exports object, providing access to the module's exports and scoped variables.

Explicit Binding: JavaScript provides methods like call, apply, and bind to explicitly set the value of this when calling a function.

Understanding this is essential for writing robust JavaScript code. By grasping the various contexts in which this operates, developers can wield its power effectively. Remember, this is not as daunting as it seems – it's just a pointer that adapts to its surroundings, guiding your functions to interact with the right objects at the right time. Happy coding!

JavaScript Tips and Tricks

Adnan Latif — Sat, 18 Nov 2023 23:15:58 +0000

JavaScript Tips and Tricks

As you know, JavaScript is the number one programming language in the world, the language of the web, of mobile hybrid apps (like React Native), of the server side (like NodeJS ) and has many other implementations. It’s also the starting point for many new developers to the world of programming, as it can be used to display a simple alert in the web browser but also to control a robot (using nodebot, or nodruino). The developers who master JavaScript and write organized and performant code have become the most sought after in the job market.

In this article, I’ll share a set of JavaScript tips, tricks and best practices that should be known by all JavaScript developers regardless of their browser/engine or the SSJS (Server Side JavaScript) interpreter.

use === instead of ==

The == (or !=) operator performs an automatic type conversion if needed. The === (or !==) operator will not perform any conversion. It compares the value and the type, which could be considered faster than ==.

undefined, null, 0, false, NaN, ‘’(empty string) are all falsy.

Find max and min from an array

The traditional way to find min and max is loop through each element of array and find min and max. Below code snippet is showing to find min and max from array using spread operator.

Merging two arrays

Repeating string multiple times

String to Number conversion

Multiple variable assignments

Multiple condition check

Remove duplicates from an array

Sum the values of an array

Convert a string to an array

Remove falsey values from an array

Alternatives of for loop

OR Short circuit

AND Short circuit

Exponent power

Count function parameters

Assign default function parameter

Check if key exists in the object

Object property assignment

Assigning object value property to a variable

Removing multiple properties from an object

Here are my two bits of contribution in JavaScript community. I hope these code snippets will help someone to learn some tips and tricks in JavaScript.

Deep Dive into Docker

Adnan Latif — Tue, 14 Nov 2023 19:44:26 +0000

What is Docker

Docker is an open-source software platform to create, deploy and manage virtualized application containers on a common operating system (OS), with an ecosystem of allied tools. Docker container technology debuted in 2013; Docker Inc. was formed to support a commercial edition of container management software and be the principal sponsor of an open-source version. Mirantis acquired the Docker Enterprise business in November 2019.

How Docker works

Docker packages, provisions, and runs containers. Container technology is available through the operating system: A container packages the application service or function with all the libraries, configuration files, dependencies, and other necessary parts and parameters to operate. Each container shares the services of one underlying operating system. Docker images contain all the dependencies needed to execute code inside a container, so containers that move between Docker environments with the same OS work with no changes.

Docker uses resource isolation in the OS kernel to run multiple containers on the same OS. This is different than virtual machines (VMs), which encapsulate an entire OS with executable code on top of an abstracted layer of physical hardware resources.

Docker was created to work on the Linux platform, but has extended to offer greater support for non-Linux operating systems, including Microsoft Windows and Apple OS X. Versions of Docker for Amazon Web Services (AWS) and Microsoft Azure are available.

Why do we need Docker?

Well, Docker is a platform for building, running, and shipping applications in a consistent manner. So, if your application works on your development machine, it can run and function the same way on other machines. If you have been developing software for a while, you’ve probably come across this situation where your application works on your development machine but doesn’t somewhere else. Can you think of three reasons why this happens? Well, this can happen if one or more files are not included as part of your deployment, so your application is not completely deployed. It’s missing something. This can also happen if the target machine is running a different version of some software that your application needs. Let’s say your application needs Node version 14, but the target machine is running Node version 9. This can also happen if the configuration settings like environment variables are different across these machines. And this is where Docker comes to the rescue.

What are Containers in Docker?

With Docker, we can easily package up our application with everything it needs and run it anywhere on any machine with Docker. So, if your application needs a given version of Node and MongoDB, all of these will be included in your Applications package. Now you can take this package and run it on any machine that runs Docker. So, if it works on your development machine, it’s going to work on your test and production machines. Now there’s more. If someone joins your team, they don’t have to spend half a day or so setting up a new machine to run your application. They don’t have to install and configure all these dependencies. They simply tell Docker to bring up your application, and Docker itself will automatically download and run these dependencies inside an isolated environment called a container and this is the beauty of Docker.

This isolated environment allows multiple applications to use different versions of some software side by side. So, one application may use Node version 14. Another application may use Node version 9. Both these applications can run side by side on the same machine without messing with each other. So, this is how Docker allows us to consistently run an application on different machines. Now, there is one more benefit here. When we’re done with this application and don’t want to work on it anymore, we can remove the application and all its dependencies in one go without Docker.

As we work on different projects, our development machine gets cluttered with so many libraries and tools that are used by different applications, and then after a while, we don’t know if we can remove one or more of these tools because we’re always afraid that we would mess up with some application with Docker, we don’t have to worry about this because each application runs with its dependencies inside an isolated environment. We can safely remove an application with all its dependencies to clean up our machine. Isn’t that great? So, in a nutshell. Docker helps us consistently build, run, and ship our applications and that’s why a lot of employers are looking for people with Docker skills these days. So if you’re pursuing a job as a software or DevOps engineer, I highly encourage you to learn Docker and learn it well.

Difference between Containers and Virtual Machines?

So, in the last paragraph, I briefly talked about containers. The container is an isolated environment for running an application. Now. One of the questions that often comes up is how our containers are different from virtual machines or VMS. Do you know the differences Well, a virtual machine as the name implies, is an abstraction of a machine or physical hardware. So, we can run several virtual machines on a real physical machine. For example, we can have a Mac, and on this Mac, we can run two virtual machines, one running Windows, and the other running Linux. How do we do that using a tool called Hyper Visor I know it’s one of those computer science names. In simple terms, the Hyper Visor is a software we use to create and manage virtual machines. There are many hypervisors available out there, like Virtual Box and VMware which are cross-platform. So, they can run on Windows or Mac. OS and Linux and Hyper-V (which is only for Windows). So, with a hyper Visor, we can manage virtual machines. Now, what is the benefit of building virtual machines Well, for us, software developers, we can run an application in isolation inside a virtual machine. So, on the same physical machine, we can have two different virtual machines. Each runs a completely different application and each application has the exact dependencies it needs.

So, application one may use node version 14 and Mongo Db Version 4, and one application may use node version nine and Mongo Db Version 3. All these are running on the same machine but in different isolated environments. That’s one of the benefits of virtual machines. However, there are several problems with this model. Each virtual machine needs a full copy of an operating system that needs to be licensed, patched, and monitored. And that’s why these virtual machines are slow to start because the entire operating system must be loaded just like starting your computer. Another problem is that these virtual machines are resource-intensive because each virtual machine takes a slice of the actual physical hardware resources like CPU memory and disk space. So, if you have eight gigabytes of memory, that memory must be divided between different virtual machines. Of course, we can decide how much memory to allocate to each virtual machine. But at the end of the day, we have a limit in terms of the number of VMS, we can run on a machine, usually a handful, otherwise we’re going to run out of hardware resources. Now let’s talk about containers, containers give us the same kind of isolation, so we can run multiple applications in isolation, but they are more lightweight. They don’t need a full operating system. In fact, all containers on a single machine share the operating system of the host. So that means we need to license, patch, and monitor a single operating system. Also, because the operating system has already started on the host, a container can start up quickly, usually in a second, sometimes less. These containers also don’t need a slice of the hardware resources on the host, so we don’t need to give them a specific number of CPU cores or a slice of memory or disk space. So, on a single host, we can run tens or even hundreds of containers side by side. So, these are the differences between containers and virtual machines.

Docker Architecture

Let’s talk about the architecture of Docker. So, you understand how it works Docker uses a client-server architecture. So, it has a client component that talks to a server component using a restful API the server also called the Docker engine sits in the background and takes care of building and running Docker containers. But technically a container is just a process, like other processes running on your computer. But it’s a special kind of process which we’re going to talk about soon Now, as I told you, unlike virtual machines, containers don’t contain a full-blown operating system. Instead, all containers on a host share the operating system of the host. Now, more accurately, all these containers share the kernel of the host. What’s the kernel? The kernel is the core of an operating system. It’s like the engine of a car. It’s the part that manages all applications as well as hardware resources like memory and CPU every operating system has its own kernel or engine. These kernels have different APIs that’s why we cannot run a Windows application on Linux because under the hood this application needs to talk to the kernel of the underlying operating system. Okay, so that means on a Linux machine, we can only run Linux containers because these containers need Linux On a Windows Machine. However, we can run both Windows and Linux containers because Windows 10 is now shipped with a custom-built Linux kernel. This is in addition to the Windows Kernel, that’s always been in Windows. It’s not a replacement. So, with this Linux kernel now we can run Linux applications natively on Windows, so on Windows, we can run both Linux and Windows containers. Our Windows containers share the Windows kernel and our Linux containers share the Linux kernel. Okay, now, what about Mac OS Well, Mac OS has its own kernel, which is different from Linux and Windows kernels, and this kernel does not have native support for continuous applications. So docker on Mac uses a lightweight Linux virtual machine to run Linux containers. Alright enough about the architecture. Next we’re going to install Docker and that’s where the fun begins.

Now the fun part begins, where we get our hands dirty with Docker

Let’s install the latest version of Docker. If you have an existing version of Docker on your machine, I highly encourage you to upgrade to the latest version because your version might be old and not compatible with the version I’m using this Quest.

I am using the version at the time of blog is 20.10.13.

Go to Docker docs to get the latest version of docker. Docker desktop is available for Windows, Mac and Linux. We have a Docker desktop which is the combination of the Docker engine, plus a bunch of other tools.

Let’s look at the instructions for Windows. You can download the latest version from Docker Hub and make sure to read system requirement. What are the things that is really important is enabling hyper-v and containers Windows features just go to the settings where you can turn on or turn off those features.

Docker Daily use Commands

docker version

This command is used to get the currently installed version of docker.

docker pull

Usage: docker pull

This command is used to pull images from the docker repository(hub.docker.com).

docker run

Usage: docker run -it -d

This command is used to create a container from an image.

-d: To start a container in detached mode, you use -d=true or just -d option. By design, containers started in detached mode exit when the root process used to run the container exits, unless you also specify the --rm option.
-it: For interactive processes (like a shell), you must use -i -t together in order to allocate a tty for the container process. -i -t is often written -it as you’ll see in later examples. Specifying -t is forbidden when the client is receiving its standard input from a pipe.

docker ps

This command is used to list the running containers.

docker ps -a

This command is used to show all the running and exited containers.

docker exec

Usage: docker exec -it bash

This command is used to access the running container.

docker stop

Usage: docker stop

This command stops a running container.

docker kill

Usage: docker kill

This command kills the container by stopping its execution immediately. The difference between ‘docker kill’ and ‘docker stop’ is that ‘docker stop’ gives the container time to shutdown gracefully, in situations when it is taking too much time for getting the container to stop, one can opt to kill it.

docker commit

Usage: docker commit

This command creates a new image of an edited container on the local system.

docker login

This command is used to login to the docker hub repository.

docker push

Usage: docker push

This command is used to push an image to the docker hub repository.

docker images

This command lists all the locally stored docker images.

docker rm

Usage: docker rm

This command is used to delete a stopped container.

docker rmi

Usage: docker rmi

This command is used to delete an image from local storage.

docker build

Usage: docker build

This command is used to build an image from a specified docker file.

Creating Our First Docker Application

Let's say we have a PHP application and want to deploy it to our staging or production server. First, we make sure we have the docker configuration script included in the root directory of the application.

Create a Dockerfile in your application

Create a file with name Dockerfile at the root of your application and include the code below to tell Docker what to do when running in the production or staging environment

FROM node:alpine
COPY . /app
WORKDIR /app
CMD node app.js

Above is a sample docker script which configures Node on a staging or production server.

Installing Docker on Staging Or Production Server

For Mac get docker here.

For Windows go here.

For Linux go here.

Running Docker

After docker is installed on the staging or production server, click on the whale icon to run docker

Deploying Your Application

Copy the application to the staging or production server and do the following

Navigate to the project directory on the terminal and create a docker image.

Run the following command in the terminal and it will create a docker image of the application and download all the necessary dependencies needed for the application to run successfully

docker build -t <name to give to your image>

Convert Docker image of the Application into a Running container.

Run the following command in terminal and it will use create a running container with all the needed dependencies and start the application.

docker run -p 9090:80 <name to give to your container>

The 9090 is the port we want to access our application on. 80 is the port the container is exposing for the host to access.

Below are some useful Docker commands

Stopping a running image

docker stop <id-of-image>

Starting an image which is not running

docker start <id-of-image>

Removing an image from docker

docker rmi <id-of-image>

Removing a container from docker

docker rm <id-of-container>

For the moment that’s enough to understand what is docker and how we can use it. In my next blog we will write a full React and Node app and run it on Docker by using

docker-compose.yml

With docker-compose.yml we don’t need to restart Docker again and again to publish the new changes. When we save the file Docker will automatically publish the new changes.

This post was originally posted on Medium.com by me. 😃

DEV Community: Adnan Latif

From 41 Minutes to 8 Minutes: How I Made Our CI/CD Pipeline 5x Faster

From 41 Minutes to 8 Minutes: How I Made Our CI/CD Pipeline 5x Faster

Introduction

The Problem

The Solutions

Reduce the size of the Docker Build Context

Dependency Caching

Improve Docker Image Handling

Run Static Analysis and Testing in Parallel

Automatic Backend Deployment

Results

Lessons Learned

Conclusion

From Confusion to Clarity: Understanding ‘this’ in JavaScript

From Confusion to Clarity: Understanding ‘this’ in JavaScript

JavaScript Tips and Tricks

JavaScript Tips and Tricks

*use *=== *instead of *==

Find max and min from an array

Merging two arrays

Repeating string multiple times

String to Number conversion

Multiple variable assignments

Multiple condition check

Remove duplicates from an array

Sum the values of an array

Convert a string to an array

Remove falsey values from an array

Alternatives of for loop

OR Short circuit

AND Short circuit

Exponent power

Count function parameters

Assign default function parameter

Object property assignment

Assigning object value property to a variable

Removing multiple properties from an object

Deep Dive into Docker

Creating Our First Docker Application

Below are some useful Docker commands

use === instead of ==