DEV Community: Lidor Ettinger

Amplify Your Tech Stack with Jenkins Shared Libraries

Lidor Ettinger — Tue, 23 May 2023 17:18:47 +0000

With the growing adoption of the Jenkins Pipeline in organizations, common patterns emerge, necessitating the sharing of reusable components to enhance code management and eliminate redundancies. Jenkins’ “Shared Libraries” feature facilitates this by allowing the definition of libraries in external repositories, seamlessly integrated into existing Pipelines.

In this post, I will guide you through the process of harnessing the full potential of shared functions in Groovy pipelines. By implementing these shared functions, you will be able to unlock scalability, maintainability, and efficiency in your pipelines, resulting in heightened productivity and accelerated development processes.

What should you expect to learn?

Achieving consistency and modularity with Shared Libraries in DevOps processes
Orchestrating pipeline workflows by modularizing components
Reusability and benefits of shared common functions for collaboration

Consistency through Modularization

By leveraging Shared Libraries, DevOps teams can modularize processes across the entire project lifecycle, enabling tasks like configuration changes, quality checks, and dependency gathering to be encapsulated within reusable library functions. This approach minimizes error risks and fosters consistency throughout the development process.

Orchestrating the Pipeline Workflow

As DevOps faces a myriad of requests for creating and customizing pipelines, becoming the central focal point, it is crucial to advance towards a more sophisticated orchestration approach. Rather than writing numerous repetitive pipelines, we can deconstruct workflow processes into modular components.

(root)
+- src
|   +- org
|       +- foo
|            +- functions
|.               +- docker.groovy
|            +- pipelines
|                +- backend.groovy
+- vars
|   +- commonPipeline.groovy
+- resources
|   +- org
|       +- foo
|           +- docker.yaml (not discussed yet)

Unveiling the skeleton pipeline

Let’s lay the foundation for a standardized skeleton pipeline that provides developers with a clear framework to follow.

Hers is the Core Pipeline Blueprint:

/vars/commonPipeline.groovy

#!/usr/bin/env groovy
def call(sharedLibrary, svcName, buildCommands, pod, slackChannel) {
...
    pipeline {
        agent {
...
        }
        stages {
            stage('Initialization') {
...
            }
            stage('Compilation') {
                when { expression { buildCommands['compileData'].run } }
                steps {
                    echo "Starting Compilation stage"
                    script {
                        try {
                            sharedLibrary.executeStage("compile", buildCommands['compileData'])
                        }  catch(Exception e) {
                            echo "Failed in compilation stage: ${e.toString()}"
                            throw e
                        }
                    }
                }
            }
            stage('Unit test') {
...
            }
            stage('Build and Upload Artifact') {
...
            }
            stage('Integration Tests') {
...
            }
            stage('Deployment') {
...
            }
        }
        post {
...
        }
    }
}

The Reusability of Common Functions

Unleashing the benefits of Shared Common Functions can foster collaboration and ensure high professional development practices.

A Repetitive Function Example for Reference:

/src/org/foo/functions/docker.groovy

def buildImage(Map args=[:]) {
    if (!args.containerName) args.containerName = containerName
    if (!args.version) args.version = "latest"
    container(args.containerName) {
        return docker.withRegistry(
                urlRegistry,
                login
            ) {
            docker.build("${args.imageName}:${args.version}")
        }
    }
}

The responsibility of the build image function is to build a docker image.

Do you ponder the frequency of using this function throughout the lifespan of our workflow?

If the answer is — yes

You might want to consider organizing the function within the common function section, ensuring it is logically structured in a hierarchical manner.

In our scenario, we have created a folder where we store shared functions related to Docker. This is where we create and store all the Docker functions for our various use cases. Subsequently, we can import the docker.groovy class and conveniently reuse all the functions it contains.

Building the pipeline workflow

The workflow class, located in the shared library, is distinct and tailored to the specific needs of the team or product. It serves as an individualized class within the library and can potentially utilize other shared classes, such as the docker.groovy mentioned earlier. Essentially, the purpose of the pipeline workflow is to incorporate and build upon the skeleton structure we previously established.

Here is an example:

/src/org/foo/pipelines/backend.groovy

package org.foo.pipelines;
import groovy.transform.Field

@Field String svcName = (scm.getUserRemoteConfigs()[0].getUrl().tokenize('/')[3].split("\\.")[0]).toLowerCase()
@Field String containerName = 'docker', organization = "naturalett"
@Field def dockerActions = new org.foo.functions.docker()

def executeStage(stageName, stageData, tag="") {
    switch (stageName) {
        case "initializaion":
            this.initializaion(stageData)
            break;
        case "compile":
            this.compile(stageData)
            break;
        case "test":
            this.test(stageData)
            break;
        case "artifact":
            this.artifact(stageData)
            break;
        case "int-test":
            this.intTest(stageData)
            break;
        case "deployment":
            this.deployment()
            break;
    }
}
def initializaion(stageData) {
    echo "TODO"
}
def compile(stageData) {
    container(containerName) {
        creds.setupCredentials()
        image = dockerActions.buildImage(
            imageName: "${organization}/${svcName}"
        )
        // Alternative way to build the image: image = docker.build("${organization}/${svcName}")
    }
}
def test(stageData) {
    echo "TODO"
}
def artifact(stageData) {
    echo "TODO"
}
def intTest(stageData) {
    echo "TODO"
}
def deployment(stageData) {
    echo "TODO"
}
def successStep() {
    echo "TODO"
}
return this

The differentiation between each pipeline occurs within the workflow itself. Each pipeline defines its own workflow and utilizes the shared classes from the shared library. This approach allows us to have a tailored workflow for each team or product while also promoting the reusability of functions.

Summarize

We’ve gained an understanding and explored the concept of Jenkins Shared Libraries and their benefits in optimizing Jenkins Pipelines. Shared Libraries enable the reuse of components across projects, enhancing code management and eliminating redundancies. By modularizing processes and encapsulating them within reusable library functions, organizations can achieve consistency and modularity in their DevOps processes.

Our next steps involve…

Furthermore, there are additional practices that justify further discussion. In the upcoming post, I will delve into further functionality regarding automated pipeline decisions, including:

Configuring project language for initial setup
Jenkinsfile implementation

Are you looking to enhance your DevOps skills and learn how to implement continuous integration using Jenkins container pipelines? Join my course, where you’ll gain practical experience in various pipeline scenarios and master production skills. Take the first step towards becoming a DevOps expert today.

Hands-On Mastering DevOps — CI and Jenkins Container Pipelines

Optimize Development with Jenkins Pipelines and Continuous Integration

Lidor Ettinger — Fri, 19 May 2023 22:42:13 +0000

Developing and releasing new software versions is an ongoing process that demands careful attention to detail. It’s essential to have the ability to track and analyze the entire process, even retrospectively, to identify any issues and take corrective measures.

This is where the concept of continuous integration comes into the picture.

By adopting a continuous integration approach, we can effectively monitor the various stages of the software development process and analyze the results. This helps us to identify any potential issues, analyze them, and make the necessary adjustments to improve the overall development process.

In this blog post, we will explore the concept of continuous integration and its benefits. We will discuss how to achieve continuous integration using Jenkins and provide insights on how this approach can help you streamline your software development process.

Want to make practice training about Continuous Integration and Jenkins?
You are invited to take a look at the course I’ve created, where you will write a complete pipeline in AWS based on SDLC principles, with my guidance.
Continuous Integration and Jenkins Pipelines in AWS

What triggers continuous integration from the perspective of the development team?

The development team triggers continuous integration by pushing code changes to the code repository. This action triggers an automated pipeline that builds, tests, and deploys the updated software version.

The concept of a structured workflow is to establish a standardized order of operations that developers agree upon, enabling them to ensure that their subsequent versions are built in accordance with the software development life cycle (SDLC) defined by management. Let’s now examine some of the primary advantages of continuous integration:

Version history — Developers can easily track their production versions and compare the performance of different versions during development. In addition, they can quickly rollback to a previous version in case of any production issues.

Quality testing — Developers can test their versions on a staging environment to demonstrate how the new version performs in an environment similar to production. Instead of running the version on their local machine, which may not be comparable to the real environment, they can define a set of tests, such as unit tests, integration tests, and more, that will take their new version through a predefined workflow. This testing process will serve as their signature and ensure that the new version is safe to be deployed in a production environment.

Schedule triggering — Developers no longer need to manually trigger their pipeline or define a new pipeline for each new project. As a DevOps team, it is our responsibility to create a robust system that will attach to each project its own pipeline. Whether it’s a common pipeline with slight changes to match the project or the same pipeline, developers can focus on writing code while the continuous integration takes care of the rest. Scheduling an automatic triggering, for example, every morning or evening, will ensure that the current code in Github is always ready for release.

Jenkins in the Age of Continuous Integration

To achieve the desired pipeline workflow, we’ll deploy Jenkins and create a comprehensive pipeline that prioritizes version control, automated testing, and triggers.

Prerequisites:

A virtual machine with a docker engine

Containerize Jenkins
We will deploy Jenkins in a Docker container in order to simplify the deployment of our CI/CD pipelines.

Let’s deploy Jenkins:

docker run -d \
        --name jenkins -p 8080:8080 -u root -p 50000:50000 \
        -v /var/run/docker.sock:/var/run/docker.sock \
        naturalett/jenkins:2.387-jdk11-hello-world

Validate the Jenkins Container:

docker ps | grep -i jenkins

Retrieve the Jenkins initial password:

docker exec jenkins bash -c -- 'cat /var/jenkins_home/secrets/initialAdminPassword'

Connect to Jenkins on the localhost

If you're looking for assistance with setting up Jenkins in a Docker container, I encourage you to visit my YouTube channel, where you'll find detailed tutorials and valuable insights. Feel free to subscribe for regular updates:

Building a Continuous Integration Pipeline

I chose to use Groovy in Jenkins pipelines because it offers a number of benefits:

Groovy is a scripting language that is easy to learn and use
Groovy offers features for writing concise, readable, and maintainable code
Groovy’s syntax is similar to Java, making it easier for Java developers to adopt
Groovy has excellent support for working with data formats commonly used in software development
Groovy provides an efficient and effective way to build robust and flexible CI/CD pipelines in Jenkins Our Pipeline Consists of Four Phases:

Phase 1 — The Agent

To ensure the code is built as expected with no incompatible dependencies, each pipeline requires a virtual environment. We create an agent (virtual environment) in a Docker container during the following phase. Since Jenkins is also running in a Docker container, we’ll mount the Docker socket to enable agent execution.

pipeline {
    agent {
        docker {
            image 'docker:19.03.12'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
        }
    }
...
...
...
}

Phase 2— The History of Versions

Versioning is an essential practice that allows developers to track changes to their code and compare software performance, enabling them to make informed decisions on whether to roll back to a previous version or release a new one. In the next phase, we create a Docker image from our code and assign it a tag based on our predetermined set of definitions.

For example: Date — Jenkins Build Number — Commit Hash

pipeline {
    agent {
...
    }
    stages {
        stage('Build') {
            steps {
                script {
                  def currentDate = new java.text.SimpleDateFormat("MM-dd-yyyy").format(new Date())
                  def shortCommit = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
                  customImage = docker.build("naturalett/hello-world:${currentDate}-${env.BUILD_ID}-${shortCommit}")
                }
            }
        }
    }
}

After completing the previous phase, a Docker image of our code was generated and can now be accessed in our local environment.

docker image | grep -i hello-world

Phase 3— The Test

Testing is a critical step to ensure that a new release version meets all functional and requirements tests. In the following stage, we run tests against the Docker image generated in the previous stage, which contains the potential next release.

pipeline {
    agent {
...
    }
    stages {
        stage('Test') {
            steps {
                script {
                    customImage.inside {
                        sh """#!/bin/bash
                        cd /app
                        pytest test_*.py -v --junitxml='test-results.xml'"""
                    }
                }
            }
        }
    }
}

Phase 4— The Scheduling Trigger

Automating the pipeline trigger allows developers to focus on code writing while ensuring stability and readiness for the next release. To achieve this, we define a morning schedule that triggers the pipeline automatically as the development team starts their day.

pipeline {
    agent {
...
    }
    triggers {
        // https://crontab.guru
        cron '00 7 * * *'
    }
    stages {
...
    }
}

An end-to-end pipeline of the process
To execute our pipeline, we have included a pre-defined pipeline into Jenkins. Simply initiate the “my-first-pipeline” Jenkins job to begin.

Agent — is a virtual environment used for the pipeline
Trigger — is used for automatic scheduling in the pipeline
Clone stage — is responsible for cloning the project repository
Build stage — involves creating the Docker image for the project to access the latest commit and other Git features, we install Git package
Test stage — involves performing tests on our Docker image

pipeline {
    agent {
        docker {
            image 'docker:19.03.12'
            args '-v /var/run/docker.sock:/var/run/docker.sock'
        }
    }
    triggers {
        // https://crontab.guru
        cron '00 7 * * *'
    }
    stages {
        stage('Clone') {
            steps {
                git branch: 'main', url: 'https://github.com/naturalett/hello-world.git'
            }
        }
        stage('Build') {
            steps {
                script {
                  sh 'apk add git'
                  def currentDate = new java.text.SimpleDateFormat("MM-dd-yyyy").format(new Date())
                  def shortCommit = sh(returnStdout: true, script: "git log -n 1 --pretty=format:'%h'").trim()
                  customImage = docker.build("naturalett/hello-world:${currentDate}-${env.BUILD_ID}-${shortCommit}")
                }
            }
        }
        stage('Test') {
            steps {
                script {
                    customImage.inside {
                        sh """#!/bin/bash
                        cd /app
                        pytest test_*.py -v --junitxml='test-results.xml'"""
                    }
                }
            }
        }
    }
}

Summarize
We’ve gained an understanding of how continuous integration fits into our daily work, as well as some key pipeline workflows, which we’ve tried out firsthand.

If you’re interested in further exploring different pipeline scenarios and learning practical production skills for implementing continuous integration in your own work, check out my course:

Hands-On Mastering DevOps — CI and Jenkins Container Pipelines